Image processing method, electronic device, readable medium and program product

Abstract

The invention relates to the technical field of image processing, in particular to an image processing method, electronic equipment, a readable medium and a program product. According to the method, in a continuous shooting mode, when a shooting event is triggered each time, a camera is controlled to collect multiple frames of images, and then after the multiple frames of images corresponding to the shooting event each time are subjected to fusion noise reduction processing, a frame of fusion image is obtained and serves as an event image corresponding to the shooting event each time; therefore, it is ensured that multiple frames of event images shot in the continuous shooting mode have high definition.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image processing method, electronic device, readable medium, and program product. Background Technology

[0002] Currently, camera functionality is a common feature provided by electronic devices such as smartphones equipped with cameras. In the camera's shooting mode, users typically use the burst mode to record the different postures of a moving subject (such as a running person) at different times. For example, a user can long-press the shutter button to trigger the burst mode of the camera app. After the smartphone or other electronic device responds to the user's operation and takes consecutive photos, the camera app can save multiple frames of event images, where each frame can be an image captured each time a photo is taken.

[0003] However, some or all of the multi-frame images captured directly in burst mode may be blurry, failing to meet the clarity requirements of users when using burst mode for the generated event images. Users who want to obtain clearer event images can only retake the shot or filter from the captured multi-frame event images, resulting in a poor user experience. Summary of the Invention

[0004] This application provides an image processing method, electronic device, readable medium, and program product that achieves high clarity in multiple frames of event images captured in continuous shooting mode.

[0005] In a first aspect, this application provides an image processing method applied to an electronic device. The method includes: detecting a burst shooting command; in response to the burst shooting command, acquiring M frames of images, wherein the burst shooting command triggers N shooting events, M > N ≥ 2; determining N reference frames and denoised frames corresponding to each reference frame from the M frames of images, wherein each reference frame corresponds to K denoised frames, K ≥ 2; and performing fusion denoising on each reference frame based on the denoised frames corresponding to each reference frame to obtain N frames of event images corresponding to the N shooting events.

[0006] For example, the aforementioned burst shooting command could be triggered by a user pressing and holding the shutter button on the camera interface. This would activate the camera application of the electronic device, allowing it to enter burst shooting mode and capture multiple photos. In this burst shooting mode, based on the method described above, the electronic device can control the camera to capture multiple frames of images each time a shooting event is triggered. These multiple frames are then fused and denoised to obtain a single fused image, which serves as the event image for each shooting event, such as the N event images corresponding to N shooting events mentioned above. This ensures that the multiple event images captured in burst shooting mode all have high clarity.

[0007] In one possible implementation of the first aspect above, determining N reference frames and corresponding denoised frames from M frame images includes: sorting the M frame images according to the shooting time to obtain a first image sequence; determining N reference frames from the first image sequence; and determining P denoising groups based on the N reference frames, wherein the P denoising groups include a first denoising group, the first denoising group includes two adjacent first and second reference frames, and L frame images between the first and second reference frames, 1≤L≤(K-1), and the denoised frames corresponding to the first and second reference frames all include L frame images.

[0008] In one possible implementation of the first aspect described above, the denoised frame of the first reference frame further includes a second reference frame, and the denoised frame of the second reference frame further includes the first reference frame.

[0009] For example, the above-described fusion noise reduction method may include arranging multiple frames (e.g., M frames) captured by the camera into an image frame sequence according to the shooting events, and selecting a number of reference frames from this sequence with equal frame intervals (e.g., equal intervals of L frames) corresponding to the number of shooting events. Then, two adjacent reference frames and the noise-reduced frames between two adjacent reference frames are combined into a noise reduction group, i.e., the aforementioned P noise reduction groups. For any one of the two reference frames within the same noise reduction group, such as the aforementioned first reference frame or second reference frame, the remaining images (including the other reference frame) within the noise reduction group can be used for fusion noise reduction processing. That is, two reference frames within a noise reduction group can share a portion of the image for fusion noise reduction, thus reducing the number of noise-reduced frames used for fusion noise reduction of two reference frames, thereby saving equipment costs.

[0010] In one possible implementation of the first aspect above, when N is odd, the P noise reduction groups also include a second noise reduction group, which includes a third reference frame and K noise reduction frames.

[0011] In one possible implementation of the first aspect described above, the third reference frame is located after (N-1) reference frames in the N-frame reference frame sequence in the first image sequence.

[0012] In some embodiments, the third reference frame may be the last reference frame among a plurality of reference frames selected from the first image sequence.

[0013] For example, if N is an odd number, and N photo events capture a total of M = N*n frames, then (N+1) / 2 noise reduction groups are determined within the image sequence composed of these M frames, i.e., P = (N+1) / 2. Each noise reduction group can include n frames, and the last frame in this group can be, for example, the last frame among the n-1 frames captured within the burst shooting interval after the triggering of the current photo event. In this case, the last noise reduction group may include only one reference frame, such as the third reference frame mentioned above. This reference frame can be an image frame captured at the triggering time of the corresponding photo event, such as the image frame captured at the triggering time of the (N-1)th photo event. Correspondingly, within the last noise reduction group, the n-1 frames captured within the burst shooting interval after the triggering of the corresponding photo event are all noise-reduced frames of the unique reference frame (e.g., the third reference frame) within that noise reduction group.

[0014] For example, when N is even, N / 2 noise reduction groups can be determined from the image sequence consisting of M (M = N*n) frames captured in N shooting events. Each noise reduction group can include n frames. The first frame in each noise reduction group can be the image frame captured at the trigger time of the corresponding shooting event, and the last frame in the noise reduction group can be the last frame of the n-1 frames captured within the continuous shooting interval after the trigger of the shooting event. Based on this, the first and last frames in each noise reduction group can be determined as a pair of reference frames.

[0015] In some embodiments, within the same noise reduction group, the first and last frames can be used as two reference frames by default, with the intra-group frame interval L = K-1 between the two reference frames. In other embodiments, within the same noise reduction group, other image frames besides the first and last frames can be selected as reference frames within that noise reduction group, in which case L can be less than (K-1).

[0016] In one possible implementation of the first aspect described above, each reference frame is the first frame of n frames captured in each shooting event, where M equals N*n. That is, the first frame in each noise reduction group can be an image frame captured at the trigger time of the corresponding shooting event, which can be the moment when image acquisition begins for each shooting event.

[0017] In one possible implementation of the first aspect described above, the interval L frames between two adjacent reference frames satisfies L equal to n-2 or L equal to n-1.

[0018] In some embodiments, when selecting a reference frame, the number of intra-group frame intervals L for each noise reduction group ultimately selected can be controlled to be equal to n-2, for example, as described below. Figures 3a to 3bThe example burst interval S-1, where the burst interval S = n-1. In other embodiments, when selecting a reference frame, the number of intra-group frame intervals L for each noise reduction group ultimately selected can be controlled to be equal to n-1, for example, as described below. Figure 4b The example continuous shooting interval S, where the continuous shooting interval S = n-1.

[0019] In one possible implementation of the first aspect described above, fusion denoising is performed on each reference frame based on the denoised frames corresponding to each reference frame, including: calculating and storing the L-frame image between the second reference frame and the first reference frame and the second reference frame, and a first registration matrix relative to the first reference frame; performing fusion denoising on the first reference frame based on the image data of the second reference frame and the L-frame image and the first registration matrix; and performing an inverse matrix transformation based on the first registration matrix to obtain a second registration matrix of the first reference frame and the L-frame image relative to the second reference frame; and performing fusion denoising on the second reference frame based on the image data of the first reference frame and the L-frame image and the second registration matrix.

[0020] It is understandable that, since image frames within each denoising group can be reused when performing fusion denoising on different reference frames, the registration matrix calculation results of the denoised frames relative to the reference frames within each denoising group can also be reused during the image registration process before fusion denoising. This reduces the computational or processing load of image processing using multi-frame fusion algorithms, thereby further saving equipment costs. For details of this image registration process, please refer to the following text. Figure 5 The descriptions and related information shown will not be repeated here.

[0021] In one possible implementation of the first aspect above, determining N reference frames and corresponding noise-reduced frames from M frame images further includes: sorting the M frame images according to the shooting time to obtain a second image sequence; determining N reference frames from the second image sequence, wherein each reference frame is the first frame image in the n frame images captured in each shooting event; and determining Q noise reduction groups based on the N reference frames, wherein the Q noise reduction groups include an adjacent third noise reduction group and a fourth noise reduction group, the third noise reduction group includes two adjacent fourth and fifth reference frames, and L frame images between the fourth and fifth reference frames, the fourth noise reduction group includes two adjacent sixth and seventh reference frames, and L frame images between the sixth and seventh reference frames, and R frame images between the fifth and sixth reference frames satisfy R ≠ L.

[0022] In other embodiments, when performing fusion noise reduction processing, when selecting reference frames from a sequence of image frames formed sequentially from multiple frames of images from multiple shooting events, different frame intervals (i.e. unequal frame intervals) can be used to select reference frames. The frame interval between two reference frames in each noise reduction group can be the same, while the frame interval between two adjacent noise reduction groups is different from the frame interval between two reference frames in each noise reduction group.

[0023] In a second aspect, this application provides an electronic device, including: one or more processors; one or more memories; the one or more memories storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the image processing method provided in the first aspect and various possible implementations of the first aspect.

[0024] Thirdly, this application provides a computer-readable medium storing instructions that, when executed on a computer, cause the computer to perform the image processing methods provided in the first aspect and various possible implementations of the first aspect.

[0025] Fourthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the image processing method provided in the first aspect and various possible implementations of the first aspect.

[0026] The beneficial effects of the second to fourth aspects mentioned above can be referred to the relevant descriptions in the first aspect and various possible implementations of the first aspect, which will not be repeated here. Attached Figure Description

[0027] Figure 1a The image shown is a schematic diagram of a camera interface.

[0028] Figure 1b The image shown is a schematic diagram of the continuous shooting interface in continuous shooting mode.

[0029] Figure 1c The image shown is a schematic diagram of a browsing interface for event images captured using burst mode.

[0030] Figure 1d The image shown is a schematic diagram of a browsing interface for viewing multi-frame event images.

[0031] Figure 2a The diagram illustrates the process of capturing N frames of event images in a continuous shooting mode.

[0032] Figure 2b The diagram shows the process of capturing N frames of event images in continuous shooting mode according to an embodiment of this application.

[0033] Figure 3a The diagram illustrates the principle of a frame selection strategy that uses the same frame interval to select reference frames for fusion and noise reduction processing. The diagram is based on the processing of 4 event images corresponding to 4 shooting events.

[0034] Figure 3b The diagram illustrates the principle of a frame selection strategy that uses the same frame interval to select reference frames for fusion noise reduction processing. The diagram is based on the processing of 5 event images corresponding to 5 shooting events.

[0035] Figure 3c The diagram illustrates the principle of a frame selection strategy that uses the same frame interval to select reference frames for fusion noise reduction processing. The diagram is based on the processing of 6 event images corresponding to 6 shooting events.

[0036] Figure 3d The diagram illustrates the principle of a frame selection strategy that uses the same frame interval to select reference frames for fusion noise reduction processing. The diagram is based on the processing of 7 event images corresponding to 7 shooting events.

[0037] Figure 4a The diagram shows the relationship between the photo-taking event corresponding to the capture of N frames of event images in continuous shooting mode and the multi-frame images captured in each photo-taking event.

[0038] Figure 4b The diagram illustrates the principle of another frame selection strategy that uses the same frame interval to select reference frames for fusion and noise reduction processing. The diagram is based on the processing of 4 event images.

[0039] Figure 4c The diagram shows a schematic of a frame selection strategy that uses different frame intervals to select reference frames for fusion and noise reduction processing.

[0040] Figure 5 The diagram shown is a schematic representation of an image processing procedure provided in an embodiment of this application.

[0041] Figure 6 The diagram shown is a schematic representation of the implementation flow of an image processing method provided in an embodiment of this application.

[0042] Figure 7a The diagram shown is a schematic representation of the principle of generating N frames of event images by using a continuous shooting interval of 3 through fusion noise reduction processing according to an embodiment of this application.

[0043] Figure 7b The diagram shown is a schematic representation of the principle of generating N frames of event images by using a continuous shooting interval of S through fusion noise reduction processing according to an embodiment of this application.

[0044] Figure 8 The figure shown is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application.

[0045] Figure 9 The diagram shown is a schematic diagram of the operating system software structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0047] It is understood that the terminal device in the embodiments of this application may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, smart TV, wearable device, tablet computer, computer with wireless transceiver function, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc.

[0048] To facilitate understanding of the solutions in the embodiments of this application by those skilled in the art, some concepts and terms involved in the embodiments of this application will be explained below.

[0049] (1) Multi-frame fusion algorithm, which is based on the idea of ​​image fusion, can merge multiple frames of images into a clearer and more accurate image. The image processing of the multiple frames to be fused by the multi-frame fusion algorithm mainly includes image preprocessing, image registration and fusion noise reduction processing.

[0050] Image registration involves aligning multiple image frames to ensure they are spatially identical. Common image registration methods include feature-point-based registration and region-based registration. Feature-point-based registration, for example, maps feature points from other image frames to feature points in a reference frame, creating a set of registered image frames relative to the reference frame. The reference frame serves as a reference reference. In the fusion and denoising stage, each image frame in this registered set can be used to denoise the reference frame, resulting in a clearer and more accurate image.

[0051] Based on this, the image frames that are registered with the reference frame and used for noise reduction of the reference frame can be referred to as noise reduction frames or usable frames.

[0052] Figures 1a to 1d This diagram illustrates interface changes during a burst shooting scenario.

[0053] refer to Figure 1a The phone 100 can detect the user's action of long-pressing the shutter button 102 on the camera interface 101 and displaying... Figure 1b The continuous shooting interface shown is 103.

[0054] refer to Figure 1b The burst shooting interface 103 can display the number of burst shots, such as "18". This number of burst shots represents the number of event images captured in the current burst shooting mode, and can be any natural number. It can be understood that the longer the user presses and holds the shutter button 102, the more burst shots will be displayed on the burst shooting interface 103. When the user stops pressing and holding the shutter button 102, they can exit the current burst shooting mode. At this point, one burst shooting process is complete. In some embodiments, the mobile phone 100 or other electronic devices can preset an upper limit for the number of event images, such as 100, 80, or 50, etc., without limitation.

[0055] Continue to refer to Figure 1b After a burst of shots is completed, the user can click the view control 104 to view the results. Correspondingly, the phone 100 can display... Figure 1c The browsing interface shown is 105.

[0056] refer to Figure 1c Corresponding to the event images captured in burst mode, the browsing interface 105 can display multi-frame markers 106. Users can click on these multi-frame markers 106 to view multiple event images. Correspondingly, the mobile phone 100 can display... Figure 1d The event image browsing interface 107 is shown.

[0057] refer to Figure 1dThe image display area 108 of the event image browsing interface 107 can display the currently selected image. The mobile phone 100 can detect the user's left and right swiping operation in the thumbnail area 109, display thumbnails of multiple event images, and switch the image displayed in the image display area 108 to the image currently indicated by the pointer 091. Each event image frame displayed in the image display area 108 can be an image frame captured by the mobile phone 100 at various moments during the continuous shooting interval, i.e., the moment each photo-taking event triggers the camera to capture an image.

[0058] However, when the subject, such as a person, is moving quickly, the multi-frame images of the event captured in burst mode may be blurry, for example... Figure 1d The image shown in area 108 contains a shadowy figure of a person. In this situation, the burst shooting function of mobile phones and other electronic devices cannot meet the user's requirement for the clarity of the event image. To obtain a clearer event image, the user can only retake the shot or select from multiple captured event images, resulting in a poor user experience.

[0059] Refer to the above Figure 1d When the mobile phone 100 is currently performing many image processing tasks, such as when other applications are running in the background to process images or videos, the edges of the image displayed in the image display area 108 by the mobile phone 100 may appear blurry. Figure 1d The example shows a blurry image that transitions from blurry to sharp. This results in a poor visual experience for the user.

[0060] It's understandable that in burst mode, a user's long press of the shutter button can trigger multiple photo capture events. (Reference) Figure 2a The image shown represents the 0th, 1st, 2nd, 3rd, ..., Nth photo-taking events. Each photo-taking event triggers the camera to capture one frame of image. For ease of description, the image captured for each photo-taking event will be referred to as an event image. That is, if the user presses and holds the shutter button, triggering N photo-taking events, then N event images will be captured. As mentioned earlier, when the subject, such as a person, is moving quickly, some event images in the N event images captured by the N photo-taking events may be blurry.

[0061] To improve the clarity of event images, this application provides an image processing method. Specifically, in continuous shooting mode, the method controls the camera to capture multiple frames of images each time a shooting event is triggered. Then, the multiple frames corresponding to each shooting event are fused and denoised to obtain a single fused image, which serves as the event image for each shooting event.

[0062] For example, suppose a user's long press of the shutter button triggers N shutter events, refer to... Figure 2b As shown, n frames (n≥2) are captured when the 0th photo-taking event is triggered, n frames are captured for the 2nd photo-taking event, ..., and n frames are captured for the Nth photo-taking event. In some embodiments, during fusion and noise reduction, one frame from the n frames can be selected as a reference frame, and the remaining n-1 frames can be used as noise reduction frames. The reference frame is then fused and noise-reduced using the n-1 noise reduction frames to obtain the event image for the corresponding photo-taking event.

[0063] It is understood that in some embodiments, the reference frame for each photo-taking event may be the first frame of an n-frame image, the nth frame, or other frames between the first and nth frames, without limitation.

[0064] Furthermore, in other embodiments, the number of image frames captured for each of the N photo-taking events can also be different. For example, in the above example, it doesn't always have to be n frames; it can be adjusted according to the movement speed of the subject. For instance, during continuous shooting, if the subject's movement speed increases, the number of image frames captured for the corresponding photo-taking event is larger; if the subject's movement speed decreases, the number of image frames captured for the corresponding photo-taking event is smaller. Correspondingly, the number of image frames used to process the event image corresponding to each photo-taking event can also be dynamically adjusted. For example, if the number of image frames captured for a photo-taking event is large, the event image corresponding to that photo-taking event can be obtained by fusing and denoising based on more image frames; if the number of image frames captured for a photo-taking event is small, the event image corresponding to that photo-taking event can be obtained by fusing and denoising based on fewer image frames. This dynamically adjusts the equipment overhead introduced by the fusion and denoising process, thereby saving equipment costs.

[0065] It can be understood that the first frame of the n frames captured at each triggering of a photo-taking event can be acquired at the moment the event is triggered, while the remaining n-1 frames can be acquired during the continuous shooting interval between the triggering of the current photo-taking event and the next triggering of the next event. The event image corresponding to each photo-taking event can be obtained by fusing and denoising the n frames acquired between the triggering of the current photo-taking event and the next triggering of the next event.

[0066] Furthermore, in some other embodiments, to further reduce equipment costs and considering that the content differences of image frames acquired by adjacent or similar shooting events in burst mode are small, during fusion noise reduction, the multi-frame images from multiple shooting events are arranged into an image frame sequence according to the shooting events. A number of reference frames corresponding to the number of shooting events are selected, and then adjacent reference frames and the noise-reduced frames between adjacent reference frames are grouped into a noise reduction group. For any one of the two reference frames within the same noise reduction group, the remaining images within the noise reduction group (including the other reference frame) can be used for fusion noise reduction. That is, two reference frames within a noise reduction group can share a portion of the image for fusion noise reduction, thereby reducing the number of noise-reduced frames used for fusion noise reduction of two reference frames, achieving the purpose of saving equipment costs.

[0067] It is understandable that, in order to save equipment costs to a large extent, the selection of reference frames in the image frame sequence can be based on the same frame interval (hereinafter referred to as equal frame interval). The frame interval can be a natural number greater than or equal to 2. For example, the frame interval can be set to the number of (n-1) frames of images captured in the continuous shooting interval between shooting events minus 1, i.e., n-2.

[0068] The following examples, with reference to the accompanying figures, illustrate the change in the number of image frames that need to be processed when performing noise reduction processing on N frames of event images captured in burst mode before and after adjusting the strategy for selecting the reference frame (hereinafter referred to as the frame selection strategy).

[0069] refer to Figure 3a As shown, taking the capture of 4 event images (i.e., N=4) as an example, assuming the preset burst shooting interval on mobile phones and other electronic devices is 3, then in this embodiment, 4 images (i.e., n=4) can be captured each time a shooting event is triggered. The process of processing the 4 event images captured in burst shooting mode through fusion noise reduction before and after adjusting the frame selection strategy can be, for example, as follows:

[0070] Before adjusting the frame selection strategy: In the 4 frames N0 / N1 / N2 / N3 captured when the 0th photo event is triggered, for example, the reference frame is N0 and the noise-reduced frames are N1 / N2 / N3; In the 4 frames N4 / N5 / N6 / N7 captured when the 1st photo event is triggered, for example, the reference frame is N4 and the noise-reduced frames are N5 / N6 / N7; In the 4 frames N8 / N9 / N10 / N11 captured when the 2nd photo event is triggered, for example, the reference frame is N8 and the noise-reduced frames are N9 / N10 / N11; In the 4 frames N12 / N13 / N14 / N15 captured when the 3rd photo event is triggered, for example, the reference frame is N12 and the noise-reduced frames are N13 / N14 / N15.

[0071] After adjusting the frame selection strategy: In the image frame sequence N0 to N15, consisting of 16 images captured by four shooting events, taking an interval of two frames as an example, N0, N3, N6, and N9 can be selected as reference frames for the four shooting events. N0 and N3 are grouped with image frames N1 and N2 within the frame interval to form a noise reduction group, and N6 and N9 are grouped with image frames N7 and N8 within the frame interval to form another noise reduction group. Thus, when performing fusion noise reduction on N0, N1 / N2 / N3 can be used as noise reduction frames; when performing fusion noise reduction on N3, N0 / N1 / N2 can be used; when performing fusion noise reduction on N6, N7 / N8 / N9 can be used; and when performing fusion noise reduction on N9, N6 / N7 / N8 can be used. During this process, image frames N4, N5, N10, N11, and N12 to N15 do not need to be processed.

[0072] Before adjusting the frame selection strategy, processing 4 event images captured in burst mode through fusion noise reduction required processing 16 images. After adjusting the frame selection strategy, processing 4 event images captured in burst mode through fusion noise reduction only requires processing 8 images.

[0073] refer to Figure 3b As shown, taking the capture of 5 event images (i.e., N=5) as an example, assuming the preset burst shooting interval on mobile phones and other electronic devices is 3, then in this embodiment, 4 images can be captured each time a shooting event is triggered (i.e., n=4). The process of processing the 5 event images captured in burst shooting mode through fusion noise reduction before and after adjusting the frame selection strategy can be, for example, as follows:

[0074] Before adjusting the frame selection strategy: Compared to the above Figure 3a The "Before Adjusting Frame Selection Strategy" option has been updated to include: among the four frames N16 / N17 / N18 / N19 captured when the fourth photo-taking event is triggered, for example, the reference frame is N16 and the noise-reduced frames are N17 / N18 / N19.

[0075] After adjusting the frame selection strategy: In the image frame sequence N0 to N19, consisting of 20 images captured by 5 photo-taking events, taking an interval of 2 frames as an example, N0, N3, N6, N9, and N12 can be selected as reference frames for the 5 photo-taking events. N0 and N3 are grouped with image frames N1 and N2 within the frame interval to form a noise reduction group; N6 and N9 are grouped with image frames N7 and N8 within the frame interval to form a noise reduction group; and N12 is grouped with image frames N13, N14, and N15 to form a noise reduction group. Thus, compared to the above... Figure 3a The "After adjusting the frame selection strategy" shown adds the following: When performing fusion and noise reduction processing on N12, N13 / N14 / N15 can be used as noise reduction frames. In this process, image frames N4, N5, N10, N11, and N16 to N19 do not need to be processed.

[0076] Before adjusting the frame selection strategy, processing 5 frames of event images captured in burst mode through fusion noise reduction required processing 20 frames. After adjusting the frame selection strategy, processing 5 frames of event images captured in burst mode through fusion noise reduction only requires processing 12 frames.

[0077] refer to Figure 3c As shown, taking the capture of 6 event images (i.e., N=6) as an example, assuming the preset burst shooting interval on mobile phones and other electronic devices is 3, then in this embodiment, 4 images (i.e., n=4) can be captured each time a shooting event is triggered. The process of processing the 6 event images captured in burst shooting mode through fusion noise reduction before and after adjusting the frame selection strategy can be, for example, as follows:

[0078] Before adjusting the frame selection strategy: Compared to the above Figure 3b The "Before Adjusting Frame Selection Strategy" section shows the addition of the following: among the four frames N20 / N21 / N22 / N23 captured when the 5th photo-taking event is triggered, for example, the reference frame is N20 and the noise-reduced frames are N21 / N22 / N23.

[0079] After adjusting the frame selection strategy: In the image frame sequence N0 to N23, consisting of 24 images captured by 6 photo-taking events, taking an interval of 2 frames as an example, N0, N3, N6, N9, N12, and N15 can be selected as reference frames for the 6 photo-taking events. N0 and N3 are grouped with image frames N1 and N2 within the frame interval to form a noise reduction group; N6 and N9 are grouped with image frames N7 and N8 within the frame interval to form a noise reduction group; and N12 and N15 are grouped with image frames N13 and N14 within the frame interval to form a noise reduction group. Thus, compared to the above... Figure 3b The "After adjusting the frame selection strategy" shown adds the following: When performing fusion and noise reduction processing on N15, N12 / N13 / N14 can be used as noise reduction frames. In this process, image frames N4, N5, N10, N11, and N16 to N123 do not need to be processed.

[0080] Before adjusting the frame selection strategy, processing 6 frames of event images captured in burst mode through fusion noise reduction required processing 24 frames. After adjusting the frame selection strategy, processing 6 frames of event images captured in burst mode through fusion noise reduction only requires processing 12 frames.

[0081] refer to Figure 3d As shown, taking the capture of 7 event images (i.e., N=7) as an example, assuming the preset burst shooting interval on mobile phones and other electronic devices is 3, then in this embodiment, 4 images can be captured each time a shooting event is triggered (i.e., n=4). The process of processing the 7 event images captured in burst shooting mode through fusion noise reduction before and after adjusting the frame selection strategy can be, for example, as follows:

[0082] Before adjusting the frame selection strategy: Compared to the above Figure 3c The "Before Adjusting Frame Selection Strategy" section shows the addition of the following: among the four frames N24 / N25 / N26 / N27 captured when the 6th photo-taking event is triggered, for example, the reference frame is N24 and the noise-reduced frames are N25 / N26 / N27.

[0083] After adjusting the frame selection strategy: In the image frame sequence N0 to N27, which consists of 28 images captured by 7 photo-taking events, taking an interval of 2 frames as an example, N0, N3, N6, N9, N12, N15, and N18 can be selected as reference frames for the 7 photo-taking events. N0 and N3 are grouped with image frames N1 and N2 within the frame interval to form a noise reduction group; N6 and N9 are grouped with image frames N7 and N8 within the frame interval to form a noise reduction group; and N12 and N15 are grouped with image frames N13 and N14 within the frame interval to form a noise reduction group. Thus, compared to the above... Figure 3c The "After adjusting the frame selection strategy" shown adds the following: When performing fusion and noise reduction processing on N18, N19 / N20 / N21 can be used as noise reduction frames. In this process, image frames N4, N5, N10, N11, N16, N17, N22 to N27 do not need to be processed.

[0084] Before adjusting the frame selection strategy, processing 7 event images captured in burst mode through fusion noise reduction required processing 28 frames. After adjusting the frame selection strategy, processing 7 event images captured in burst mode through fusion noise reduction only requires processing 16 frames.

[0085] Similarly, it can be seen that when N is an even number, adjusting the frame selection strategy can halve the number of image frames required for processing N frames of event images through fusion and noise reduction, resulting in a 1 / 2 reduction in equipment costs. When N is an odd number, adjusting the frame selection strategy can reduce the number of image frames required for processing N frames of event images through fusion and noise reduction by approximately 40%, resulting in a 40% reduction in equipment costs.

[0086] In other embodiments, taking an image with a frame interval of 3 frames as an example, refer to... Figure 4a As shown, the image frames captured for the 0th, 1st, 2nd, 3rd, ..., Nth photo events are selected as reference frames. S (e.g., 3) image frames captured within the first burst interval between the 0th and 1st photo events are repeatedly used as noise reduction frames for the reference frames corresponding to the 0th and 1st photo events. Similarly, S (e.g., 3) image frames captured within the third burst interval between the 2nd and 3rd photo events are repeatedly used as noise reduction frames for the reference frames corresponding to the 2nd and 3rd photo events, and so on.

[0087] Correspondingly, each photo-taking event triggers the process of generating a corresponding event image, which can be referred to as... Figure 4b As shown, for the 0th photo, the reference frame is N0, and the noise reduction frames are N1 / N2 / N3; for the 1st photo, the reference frame is N4, and the noise reduction frames are N1 / N2 / N3; for the 2nd photo, the reference frame is N8, and the noise reduction frames are N9 / N10 / N11; for the 3rd photo, the reference frame is N11, and the noise reduction frames are N9 / N10 / N11; and so on.

[0088] In this way, at least the computational load of performing the aforementioned BPS processing and registration calculations on multiple frames captured within a certain continuous shooting interval can be reduced. For example, the computational load on image frames captured within the second continuous shooting interval (e.g.) can be reduced. Figure 4b The image processing computation of N5, N6, N7, N13, N14, and N15 (shown) is also beneficial to save equipment costs to a certain extent.

[0089] In other embodiments, when performing fusion noise reduction processing, when selecting reference frames from a sequence of image frames formed sequentially from multiple frames of images from multiple shooting events, different frame intervals (i.e. unequal frame intervals) can be used to select reference frames. The frame interval between two reference frames in each noise reduction group can be the same, while the frame interval between two adjacent noise reduction groups is different from the frame interval between two reference frames in each noise reduction group.

[0090] For example, the first frame of the n frames from the 0th photo event can be selected as the reference frame for the 0th photo event, and the remaining n-1 frames can be used as noise reduction frames. During the fusion and noise reduction process to obtain the event image of the 1st photo event, the nth frame of the n frames from the 0th photo event can be selected as the reference frame for the 1st photo event, and the remaining n-1 frames can be used as noise reduction frames. Similarly, during the fusion and noise reduction process to obtain the event image of the 2nd photo event, the first frame of the n frames from the 2nd photo event can be selected as the reference frame for the 2nd photo event, and the remaining n-1 frames can be used as noise reduction frames. Similarly, during the fusion and noise reduction process to obtain the event image of the 3rd photo event, the nth frame of the n frames from the 2nd photo event can be selected as the reference frame for the 3rd photo event, and the remaining n-1 frames can be used as noise reduction frames.

[0091] Similarly, if N is odd, and N photo events result in M ​​= N*n frames, then (N+1) / 2 noise reduction groups are determined within the image sequence of these M frames. Each noise reduction group can include n frames. The first frame in each noise reduction group can be the image frame captured at the trigger moment of the corresponding photo event, which can be the moment when image acquisition begins for each photo event. The last frame in the noise reduction group can be the last frame among the n-1 frames captured within the continuous shooting interval after the trigger of the photo event. Based on this, the first and last frames in each noise reduction group can be determined as a pair of reference frames. It should be noted that in this case, the last noise reduction group determined in the above image sequence may include only one reference frame. This reference frame can be the first frame image captured in the corresponding photo event, such as the first frame image captured in the N-1th photo event. Correspondingly, within the last noise reduction group, the n-1 frames captured within the continuous shooting interval after the trigger of the corresponding photo event are all noise-reduced frames of the unique reference frame within that noise reduction group.

[0092] If N is even, then N / 2 noise reduction groups can be determined from the image sequence consisting of M (M = N*n) frames captured in N shooting events. Each noise reduction group can include n frames. The first frame in each noise reduction group can be the image frame captured at the trigger time of the corresponding shooting event, and the last frame in the noise reduction group can be the last frame of the n-1 frames captured within the continuous shooting interval after the trigger of the shooting event. Based on this, the first and last frames in each noise reduction group can be determined as a pair of reference frames.

[0093] refer to Figure 4c As shown, taking N=4 and n=4 as an example, in an image sequence consisting of M=4*4=16 frames from 4 shooting events, two noise reduction groups can be identified, for example... Figure 4c The diagram shows two noise reduction groups: one consisting of four frames (N0 to N3) and the other consisting of four frames (N8 to N11). In N0 to N3, N0 is the image frame captured at the trigger time of the 0th photo-taking event (i.e., the first frame). N1 to N3 are three frames captured during the continuous shooting interval after the 0th photo-taking event. Within this noise reduction group, N0 and N3 can be identified as a pair of reference frames. For reference frame N0, the noise reduction frames used for fusion noise reduction processing can be N1 / N2 / N3, and the resulting fused image can be used as the event image for the 0th photo-taking event. For reference frame N3, the noise reduction frames used for fusion noise reduction processing are N0 / N1 / N2, and the resulting fused image can be used as the event image for the 1st photo-taking event.

[0094] Similarly, in N8 to N11, N8 is the image frame captured at the moment the second photo-taking event is triggered, and N9 to N11 are three frames captured within the continuous shooting interval after the second photo-taking event is triggered. Within this noise reduction group, N8 and N11 can be identified as a pair of reference frames. For reference frame N8, the noise reduction frames used for fusion noise reduction processing can be N9 / N10 / N11, and the fused image obtained from the corresponding processing can be used as the event image of the second photo-taking event; for reference frame N11, the noise reduction frames used for fusion noise reduction processing can be N8 / N9 / N10, and the fused image obtained from the corresponding processing can be used as the event image of the third photo-taking event.

[0095] Thus, in the process of obtaining 4 event images through fusion and noise reduction, the processing of 8 images (N4 to N7 and N12 to N15) can be reduced, which can also save equipment costs to a certain extent.

[0096] It is understandable that after selecting a number of reference frames corresponding to the number of photo events, and then forming denoising groups from two adjacent reference frames and the denoised frames between two adjacent reference frames, a pair of reference frames can be reselected within each denoising group. These reselected reference frames can be any pair of image frames within each denoising group, without restriction. That is, the reference frames ultimately used in each denoising group for fusion and denoising to obtain the event image can be different from those described above. Figures 3a to 3d , Figures 4b to 4c The example reference frame is not limited here.

[0097] In addition, in the above Figures 3a to 3d After adjusting the frame selection strategy in the example, since the image frames in each noise reduction group can be reused when performing fusion noise reduction processing on different reference frames, the calculation results of the registration matrix of the noise reduction frames in each noise reduction group relative to the reference frames can also be reused during the image registration process before fusion noise reduction processing. In this way, the amount of calculation or processing in the image processing process using multi-frame fusion algorithm can be reduced, thereby further saving equipment costs.

[0098] Refer to the above Figure 3a The example shown is a 4-frame event image processed by fusion and noise reduction. Figure 5 This diagram illustrates the image processing procedure for processing multi-frame event images using a multi-frame fusion algorithm.

[0099] refer to Figure 5 The reference frame used for the event image of the 0th photo event is N0, and the noise reduction frames are N1 / N2 / N3. The corresponding image processing steps may include:

[0100] Image registration: This involves calculating the registration matrices N1 / N2 / N3 relative to N0, based on N0. Before image registration, mobile devices and other electronic devices run relevant image preprocessing algorithms to preprocess each image frame. This preprocessing may include using a Bayer processing segment (BPS) to perform bad pixel removal, phase detection autofocus, de-mosaicing, downsampling, HDR processing, and Bayer hybrid noise reduction.

[0101] Furthermore, in the embodiments of this application, the registration matrix N1 / N2 / N3 relative to N0 calculated by the image registration process can be saved for use in the process of generating the event image of the first photo event, as detailed in the relevant description below.

[0102] Fusion denoising: For example, the calculated registration matrix, the denoised frames N1 / N2 / N3, and the reference frame N0 to be processed are input into the fusion denoising module to execute the fusion denoising processing algorithm and obtain a clearer image frame.

[0103] Image post-processing includes the use of lighting and shadow processing techniques such as local tone mapping to adjust local tones and other image details in the image frames obtained from the fused and denoised image processing. Ultimately, this process can generate event image 0 for the 0th photographing event.

[0104] It is understood that the other image processing procedures mentioned above can be executed by [the processor / processor]. The BPS is a hardware component primarily used for processing photographic image data, and its functions may include bad pixel removal, phase detection autofocus, depixelation, downsampling, HDR processing, and Bayer hybrid noise reduction. In other embodiments, the other image processing procedures mentioned above can also be performed by corresponding processors such as graphics processing units (GPUs) and image signal processors (ISPs) executing corresponding image processing algorithms; this is not limited here.

[0105] Continue to refer to Figure 5 As mentioned above, in this embodiment of the application, the reference frame used to process the event image of the first photo-taking event is N3, and the noise reduction frames are N0 / N1 / N2. The corresponding image processing procedure may also include:

[0106] Image registration: Transform the registration matrices N1 / N2 / N3 to generate registration matrices N0 / N1 / N2 relative to N3. Before this image registration, mobile phones and other electronic devices can directly reuse the N0 / N1 / N2 / N3 preprocessed by BPS or similar image preprocessing, meaning image registration can be performed without further image preprocessing of other image frames.

[0107] Furthermore, the registration matrix used in this image registration process can be obtained by transforming the registration matrices of N1 / N2 / N3 relative to N0, based on the aforementioned saved registration matrices. In some embodiments, this inverse matrix transformation can include methods such as the adjoint method, elementary row operations, Gauss-Jordan elimination, and LU decomposition, and is not limited here. Typically, the computational or processing complexity of the aforementioned inverse matrix transformation method is much smaller than the computational complexity of calculating the registration matrices of N0 / N1 / N2 relative to N3.

[0108] Fusion denoising: For example, the calculated registration matrix, the denoised frames N1 / N2 / N3, and the reference frame N0 to be processed are input into the fusion denoising module to execute the fusion denoising processing algorithm and obtain a clearer image frame.

[0109] Image post-processing includes the process of adjusting local tones and other image details in the image frame obtained from the fused and denoised image using lighting and shadow processing techniques such as local tone mapping. Ultimately, this process generates event image 1 for the first photographing event.

[0110] Similarly, the image registration process for generating the event image of the third photograph can use a registration matrix obtained by inverse matrix transformation based on the registration matrix N7 / N8 / N9 relative to N6 corresponding to the second photograph. When generating more event images, the image registration process for generating the event image of the fifth photograph can use a registration matrix obtained by inverse matrix transformation based on the registration matrix N13 / N14 / N15 relative to N12 corresponding to the fourth photograph. And so on.

[0111] Analysis of the process of generating event images for the 0th and 1st photo capture events reveals that, based on the image processing method provided in this application, the continuous shooting mode generates two consecutive photos corresponding to continuous shooting. This allows for the reuse of BPS processing results from related shooting events and multiple frames captured during the continuous shooting interval. Furthermore, the calculated registration matrix can be reused after inverse matrix transformation. This reduces the computational load of related image processing, thus improving the BPS processing, registration calculation, and other image processing performance of mobile phones and other electronic devices. For example, in some implementation scenarios, by optimizing the three registration calculations through registration matrix transformation, the image processing performance of electronic devices can be reduced from 80ms to 2ms.

[0112] The following section will describe the specific implementation process of the image processing method provided in this application, using a detailed flowchart.

[0113] It should also be stated that the steps in the methods and processes in this application are numbered for ease of reference, not to limit the order of steps. If there is an order between the steps, the textual description shall prevail.

[0114] As an example, Figure 6 An implementation flowchart of an image processing method is shown according to an embodiment of this application.

[0115] It is understood that each step in this implementation process can be performed by an electronic device such as a mobile phone, for example, by a processor or combination of processors such as a GPU or ISP configured in the mobile phone or other electronic device to execute the relevant algorithms or instructions. To avoid repetition, the executing entity of each step will not be described again when introducing the execution content of each step below.

[0116] Specifically, such as Figure 6 As shown, the implementation process may include the following steps.

[0117] S601: Continuous shooting command detected.

[0118] For example, when the mobile phone 100 detects a shooting operation instructing the camera to enter burst shooting mode, it can detect the aforementioned burst shooting command accordingly. (Refer to the above...) Figures 1a to 1b The mobile phone 100 detects a user instruction to the camera to enter burst mode, such as a user long-pressing the shutter button 102 on the camera interface 101. This operation can trigger the mobile phone 100 to display... Figure 1b The continuous shooting interface 103 shown can display to the user the number of shots taken in a burst corresponding to the duration of pressing and holding the shutter button, such as 18 or other numbers, without limitation.

[0119] S602: In response to a continuous shooting command, triggers N shooting events and captures M frames of images.

[0120] For example, a user's long press of the shutter button triggers a burst shooting operation, which corresponds to a burst shooting command. This can trigger multiple shooting events on the camera, such as the aforementioned N shooting events. The event images corresponding to these N shooting events are denoted as the number of burst shots taken by the user's burst shooting operation instruction. Each time a shooting event is triggered, the camera of the mobile phone 100 can capture n frames of images. Therefore, the M frames captured after triggering N shooting events can satisfy M = N*n. In electronic devices such as the mobile phone 100, for the burst shooting mode of the camera application, the number of image frames captured within the burst shooting interval can be preset, denoted as S, where S can be equal to n-1, or n can be equal to S+1. Based on this burst shooting interval S, the i-th event image can be obtained using S+1 frames, where i belongs to [0, N-1].

[0121] In some embodiments, the mobile phone 100 may preset the burst shooting interval to capture 3 frames of images, or preset the burst shooting interval to capture 4 or 5 frames of images, etc., without limitation.

[0122] In continuous shooting mode, the mobile phone 100 can select reference frames and noise-reducing frames from the S+1 frames captured in each shooting event and the corresponding continuous shooting interval, based on the image processing method provided in this application, and process and generate N frames of event images captured by the user in continuous shooting mode. For example, for the i-th event image to be generated, reference frames and noise-reducing frames can be selected based on the corresponding S+1 frames, where i belongs to [0, N-1]. The specific frame selection process can be referred to the relevant description in step S603 below, and will not be repeated here.

[0123] S603: Based on the first frame selection strategy, determine the N reference frames corresponding to the N shooting events from the M frame images, and the noise reduction frames corresponding to each reference frame.

[0124] For example, corresponding to the i-th frame of the event image to be generated, a reference frame and a noise-reducing frame can be selected based on the corresponding S+1 captured frame image to further improve the clarity and quality of the final generated event image through the multi-frame fusion algorithm. Taking the selection of reference frames from the image sequence with equal frame intervals as an example, N reference frames corresponding to N shooting events are determined from M frame images, and noise-reducing frames corresponding to each reference frame are selected. That is, 1 reference frame and S-1 noise-reducing frames are selected from the S+1 or n frame images captured in each shooting event. In other embodiments, the image frame captured at the trigger time of each shooting event can also be used as the reference frame, and the S frame images captured in the continuous shooting interval between two consecutive shooting events can be used as the noise-reducing frames. The specific number of noise-reducing frames selected is not limited here.

[0125] Thus, every two reference frames and the S or S-1 denoised frames between these two reference frames can form a denoising group. Each reference frame in this denoising group can use another reference frame and each denoised frame as a denoised frame for fusion denoising processing.

[0126] Specifically, N reference frames and corresponding noise-reducing frames are determined from M frames of images. First, the M frames are sorted according to their capture time to obtain a first image sequence. Then, N reference frames are determined from this first image sequence. Within the same noise reduction group, adjacent reference frames can be spaced L frames apart, where 1 ≤ L ≤ (K-1), and L is the intra-group frame interval, which can include, for example, S or S-1 as described above. Further, P noise reduction groups can be determined based on the N reference frames. Any of these P noise reduction groups, such as the first noise reduction group, can include two adjacent reference frames and L frames between adjacent reference frames within the group, i.e., the intra-group frame interval L. In other words, within each noise reduction group, one reference frame can be fused and noise-reduced using another reference frame and the intra-group frame interval L frames as noise-reducing frames.

[0127] For ease of description, in some embodiments, the number of noise-reduced frames used for fusion noise reduction processing of any reference frame within each noise reduction group can also be described as K. For every two reference frames and the L noise-reduced frames between those two reference frames, K = L + 1, and the corresponding L = K - 1. Combined with the above description of the burst shooting interval S, it can be understood that in some embodiments, L can be equal to S (e.g., Figure 4a and Figure 4b In other embodiments, L may also be equal to S-1 (e.g., Figures 3a to 3d as well as Figure 4c For the case where L = S, K = S + 1; for the case where L = S - 1, K = (S - 1) + 1 = S. Combining this with the description above regarding the ability to collect n frames of images between each photo-taking event and the next, where n = S + 1, it can also be understood that for the case where L = S, L can be equal to n - 1; for the case where L = S - 1, L can be equal to n - 2, without any restrictions.

[0128] As an example, the first frame selection strategy used in selecting the reference frame and the noise reduction frame can be either the frame selection strategy before adjustment based on the image processing method provided in this application, such as the one described above. Figures 3a to 3d The frame selection strategy before adjustment (i.e., "before adjusting the frame selection strategy") exemplified in this application can also adopt the frame selection strategy adjusted by the image processing method provided in this application, such as the one described above. Figures 3a to 3dThe example illustrates the adjusted frame selection strategy (i.e., "after adjusting the frame selection strategy"). Based on this adjusted strategy, electronic devices such as mobile phones can improve the clarity and quality of event images using multi-frame fusion algorithms while having little or no impact on image processing performance. Both of these factors contribute to improving the user's photography experience.

[0129] Refer to the above Figure 3a The example shown illustrates the adjusted frame selection strategy. Taking S=3 (corresponding to n=4) as an example, when i=0, N0 (or N0) can be selected as the reference frame, and N1, N2, N3 (or N1 / N2 / N3) as the noise reduction frames. When i=1, N3 (or N3) can be selected as the reference frame, and N0, N1, N2 (or N0 / N1 / N2) as the noise reduction frames. When i=2, N6 (or N6) can be selected as the reference frame, and N7, N8, N9 (or N7 / N8 / N9) as the noise reduction frames. When i=3, N9 (or N9) can be selected as the reference frame, and N6, N7, N8 (or N6 / N7 / N8) as the noise reduction frames. (Continue with the example...) Figures 3b to 3d As shown, and so on.

[0130] In other embodiments, for cases where the number of event images to be captured and / or the burst interval differ as instructed by the user, please refer to the following: Figures 7a to 7b The general frame selection strategy exemplified here for selecting the reference frame and the noise reduction frame can be found in the relevant description below, and will not be repeated here.

[0131] S604: Perform image preprocessing on N reference frames and the corresponding denoised frames, calculate the registration matrix of the denoised frame relative to each reference frame in the same denoising group, and perform image registration. Perform image preprocessing on the S+1 frame image, calculate the registration matrix of the S denoised frame relative to the reference frame, and perform image registration.

[0132] For example, based on the reference frame and denoised frame selected in step S603 above, reusable image frames that require image preprocessing (e.g., BPS processing) and registration calculations can be determined. For instance, image preprocessing can be performed on S+1 or n frames, and the registration matrix of S or n-1 denoised frames relative to the reference frame can be calculated for image registration. For example, as described above... Figure 3a The image frames shown are N0, N1, N2, N3, and N6, N7, N8, N9, etc.

[0133] Furthermore, during image registration, while reusing some image frames, the registration matrix calculated during the image registration process can also be reused, as described above. Figure 5As shown, for example, the registration matrix of N0 / N1 / N2 relative to N3 can be obtained by inverse matrix transformation based on the previously calculated and saved registration matrix of N1 / N2 / N3 relative to N0. Similarly, the registration matrix of N6 / N7 / N8 relative to N9 can be obtained by inverse matrix transformation based on the previously calculated and saved registration matrix of N7 / N8 / N9 relative to N6.

[0134] Thus, based on the frame selection strategy implemented in step S603, when generating a specified number of event images, electronic devices such as mobile phone 100 can reduce the processing of some image frames during step S604, thereby improving image processing performance. For example, taking the generation of 4 event images as an example, based on the adjusted frame selection strategy implemented in step S603, the processing of the aforementioned... Figure 3a The processing of the eight frames N5, N6, N10, N11 and N12 to N15 shown can improve the image processing performance of electronic devices such as mobile phones by at least 50% compared to before the frame selection strategy was adjusted.

[0135] S605: Based on the noise reduction frames corresponding to each reference frame, perform fusion noise reduction processing on each reference frame.

[0136] For example, based on the registration matrix and the preprocessed S-frame denoised frames, a multi-frame fusion algorithm is used to perform fusion denoising on the reference frames. Referring to the above... Figure 5 The example image processing procedure involves inputting the image data of each denoised frame after image preprocessing, such as the registration matrix obtained by calculation or inverse matrix transformation and BPS processing, into the fusion and denoising module for each event image to be generated. The multi-frame fusion algorithm is then used to perform fusion and denoising processing on the reference frame, resulting in a clearer and higher-quality fused image.

[0137] Then, the fused image is further processed in step S606, using lighting and shadow processing techniques such as local tone mapping to perform some image post-processing, ultimately generating the corresponding event image, for example... Figure 5 The event image shown is 0, event image 1, etc.

[0138] S606: Perform image post-processing on the fused image obtained from the processing to obtain N frames of event images corresponding to N shooting events.

[0139] For example, post-processing the fused image of the i-th frame can yield an event image of the i-th frame, where i belongs to [0, N-1]. This event image of the i-th frame can be, for example, the one described above. Figure 5 The event image shown is 0, event image 1, etc.

[0140] Based on the implementation process of steps S601 to S606 above, the image processing method provided by this application can both improve the clarity and quality of event images and ensure that the image processing performance of mobile phones and other electronic devices is not reduced due to the use of multi-frame fusion algorithms to process event images, which is beneficial to improving the user's shooting experience when using burst mode.

[0141] The following section, in conjunction with the accompanying drawings, will provide a detailed explanation of the principle behind the general frame selection strategy mentioned in step S603 above, which is used to select reference frames and noise-reducing frames.

[0142] As an example, Figure 7a According to an embodiment of this application, a schematic diagram is shown of the principle of generating N frames of event images by fusion noise reduction processing with a continuous shooting interval of 3.

[0143] like Figure 7a As shown, the number of consecutive shots, N frames, indicates that the number of event images obtained through multi-frame fusion algorithms is N frames. Taking a 3-frame consecutive shot interval as an example, the camera can capture 3 frames within the consecutive shot interval based on a preset acquisition frequency. In response to the user's consecutive shot operation to obtain N frames of event images, it can trigger the acquisition of 4*(N-1)+1 frames. It can be understood that the 4*(N-1)+1 frames mentioned above can be the raw images captured by the camera, such as raw image data in YUV format, or image frames that have undergone preliminary processing by some preprocessing algorithms; there are no restrictions here.

[0144] Based on the adjusted frame selection strategy used in the image processing method provided in this application, the following index path can be used to determine the reference frame and the noise reduction frame to be selected:

[0145] The index path of the reference frame for the i-th event image (i belongs to [0, N-1]) is:

[0146] IndexRef_i = i * 3, that is Figure 7a The N shown i*3 ;

[0147] At this point, the index paths corresponding to the 3 noise-reduced frames used to generate the i-th frame image are as follows:

[0148]

[0149] Where IndexRef_i+1 is Figure 7a The N shown i*3+1 IndexRef_i+2 is... Figure 7a The N shown i*3+2 IndexRef_i+3 is... Figure 7a The N shown i*3+3IndexRef_i-1 is... Figure 7a The N shown i*3-1 IndexRef_i-2 is... Figure 7a The N shown i*3-2 IndexRef_i-3 is... Figure 7a The N shown i*3-3 .

[0150] refer to Figure 7a As shown, when i is even, it is used for reference frame N. i*3 The number of denoised frames that undergo fusion and denoising processing can be N. i*3+1 N i*3+2 N i*3+3 When i is odd, it is used for reference frame N. i*3 The number of denoised frames that undergo fusion and denoising processing can be N. i*3-1 N i*3-2 N i*3-3 .

[0151] Taking i=2 as an example, the noise reduction frames used to perform fusion noise reduction processing on the reference frame N6 can be N7, N8, and N9.

[0152] Taking i=3 as an example, the noise reduction frames used to perform fusion noise reduction processing on the reference frame N6 can be N8, N7, and N6.

[0153] Similarly, taking i=0 as an example, the noise reduction frames used for fusing and denoising the reference frame N0 can be N1, N2, and N3; taking i=1 as an example, the noise reduction frames used for fusing and denoising the reference frame N3 can be N2, N1, and N0.

[0154] As another example Figure 7b A schematic diagram of a frame selection strategy is shown, taking an interval of S frames and a number of N frames in a burst as an example.

[0155] like Figure 7b As shown, assuming a user's long-press of the shutter button triggers N shutter events, then n frames (n≥2) are captured during the 0th shutter event, n frames are captured during the 2nd shutter event, ..., and so on, resulting in an N*n frame sequence. The number of image frames acquired during the burst interval between two adjacent shutter events is S = n-1. Based on this, during the fusion and noise reduction process to obtain the event image of the i-th (i belongs to [0, N-1]) shutter event, the index parameter of the reference frame in the aforementioned N*n frame sequence can be:

[0156] IndexRef_i=i*S=i*(n-1), for example Figure 7b The N shown i*S ;

[0157] The index parameter of the denoised frame in the above N*n frame image frame sequence, after performing fusion and denoising on the reference frame, can be:

[0158]

[0159] Where IndexRef_i+1 is Figure 7b The N shown i*S+1 IndexRef_i+2 is... Figure 7b The N shown i*S+2 IndexRef_i+3 is... Figure 7b The N shown i*S+3 IndexRef i +S means Figure 7b The N shown i*S+S ;

[0160] IndexRef_i-1 is... Figure 7b The N shown i*S-1 IndexRef_i-2 is... Figure 7b The N shown i*S-2 IndexRef_i-3 is... Figure 7b The N shown i*S-3 IndexRef i -S means Figure 7b The N shown i*S-S .

[0161] Thus, in the process of obtaining event images from multiple shooting events through fusion and noise reduction, the number of image frames processed can be reduced; for example, at least the number of frames processed can be reduced. Figure 7b The processing procedures for image frames such as Ni*S+S+1 (i is even), ..., Ni*S+S+(S-1) (i is even), N10, N11, or Ni*S+1 (i is odd), ..., Ni*S+(S-1) (i is odd) shown are beneficial for saving equipment costs.

[0162] Figure 8 A schematic diagram of the hardware structure of an electronic device is shown according to an embodiment of this application. In this embodiment, the electronic device may be, for example, the mobile phone 100 described above. In other embodiments, the electronic device may also be a tablet computer, a watch, or other similar device, which will not be elaborated here.

[0163] Taking mobile phone 100 as an example, such as Figure 8As shown, the mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identity module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0164] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile phone 100. In other embodiments of this application, the mobile phone 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0165] Processor 110 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0166] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0167] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the aforementioned memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0168] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a universal serial bus (USB) interface, etc.

[0169] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the mobile phone 100 to take pictures. The processor 110 and the display screen 194 communicate via the DSI interface to enable the mobile phone 100 to display.

[0170] USB port 130 is a USB standard compliant interface, which can be a Mini USB port, Micro USB port, USB Type-C port, etc. USB port 130 can be used to connect a charger to charge mobile phone 100, and can also be used for data transfer between mobile phone 100 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0171] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the mobile phone 100. In other embodiments of this application, the mobile phone 100 may also adopt different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0172] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the mobile phone 100. While charging the battery 142, the charging management module 140 can also supply power to the electronic device via the power management module 141.

[0173] The wireless communication function of mobile phone 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0174] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0175] Mobile communication module 150 can provide wireless communication solutions including 2G / 3G / 4G / 5G for use on mobile phone 100. Wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) for use on mobile phone 100.

[0176] In some embodiments, antenna 1 of mobile phone 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling mobile phone 100 to communicate with networks and other devices via wireless communication technology. The aforementioned wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The aforementioned GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0177] The mobile phone 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0178] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Mini-LED, a Micro-LED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0179] The mobile phone 100 can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0180] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits this electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0181] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, mobile phone 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0182] In this embodiment, the ISP of the mobile phone 100 can acquire raw image data captured by the camera 193, or acquire image data selected as reference frames or noise reduction frames from the raw image data captured by the camera 193 and processed by image processing such as BPS. Then the ISP can execute the image processing method provided in this application, use the multi-frame fusion algorithm to improve the clarity of the event image captured in burst mode, and at the same time ensure that the image processing performance of the mobile phone and other electronic devices will not be reduced after using the multi-frame fusion algorithm, thereby improving the user's shooting experience in burst mode.

[0183] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when a mobile phone 100 is selecting a frequency, the DSP performs Fourier transforms on the frequency energy.

[0184] Video codecs are used to compress or decompress digital video. Mobile phone 100 can support one or more video codecs. Thus, mobile phone 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0185] The external storage interface 120 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the mobile phone 100. The external storage card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0186] Internal memory 121 can be used to store computer executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of mobile phone 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of mobile phone 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0187] The mobile phone 100 can achieve audio functions such as music playback and recording through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0188] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0189] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Mobile phone 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, mobile phone 100 detects the intensity of the touch operation based on pressure sensor 180A. Mobile phone 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands.

[0190] The gyroscope sensor 180B can be used to determine the motion attitude of the mobile phone 100. In some embodiments, the gyroscope sensor 180B can determine the angular velocity of the mobile phone 100 around three axes (i.e., the x, y, and z axes). The gyroscope sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the mobile phone 100's shake, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the mobile phone 100 through reverse movement, thus achieving image stabilization. The gyroscope sensor 180B can also be used in navigation and motion-sensing gaming scenarios.

[0191] The 180E accelerometer can detect the magnitude of acceleration of the mobile phone 100 in various directions (typically three axes). When the mobile phone 100 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and applied to applications such as screen orientation switching and pedometers.

[0192] A distance sensor 180F is used to measure distance. The mobile phone 100 can measure distance via infrared or laser. In some embodiments, during a shooting scenario, the mobile phone 100 can utilize the distance sensor 180F to measure distance for fast focusing.

[0193] The proximity sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The mobile phone 100 emits infrared light outward through the LED. The mobile phone 100 uses the photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the mobile phone 100. When insufficient reflected light is detected, the mobile phone 100 can determine that there is no object near the mobile phone 100. The mobile phone 100 may use the proximity sensor 180G to detect when the user holds the mobile phone 100 close to their ear for a call, so as to automatically turn off the screen to save power.

[0194] The ambient light sensor 180L is used to detect ambient light levels. The phone 100 can adaptively adjust the brightness of its display 194 based on the detected ambient light. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking photos. The ambient light sensor 180L can also work in conjunction with the proximity sensor 180G to detect whether the phone 100 is in a pocket, preventing accidental touches.

[0195] The fingerprint sensor 180H is used to collect fingerprints. The phone 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, app access lock, fingerprint photography, fingerprint answering of calls, etc.

[0196] Touch sensor 180K, also known as a "touch device," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touchscreen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of mobile phone 100, in a different position than display screen 194.

[0197] Keypad 190 includes a power button, volume buttons, etc. Keypad 190 can be a mechanical keypad or a touch keypad. Mobile phone 100 can receive keypad input and generate key signal inputs related to user settings and function control of mobile phone 100.

[0198] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0199] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0200] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact and separation with the mobile phone 100. In some embodiments, the mobile phone 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the mobile phone 100 and cannot be separated from it.

[0201] Figure 9 A schematic diagram of the operating system software structure of an electronic device is shown according to an embodiment of this application. The operating system software structure will be further described below using a mobile phone 100 as an example.

[0202] It is understood that the operating system of mobile phone 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the system software structure of mobile phone 100.

[0203] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, Android... TM The system is divided into four layers, from top to bottom: application layer, application framework layer, and Android. TM runtime (Android) TM runtime and system libraries, as well as the kernel layer.

[0204] like Figure 9 As shown, the application layer can include a series of application packages. These application packages can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0205] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0206] The application framework layer may include a window manager, content provider, view system, camera service, phone manager, resource manager, notification manager, etc.

[0207] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0208] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, etc.

[0209] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0210] The camera service is used to respond to user operations detected by the camera application, such as the user pressing and holding the shutter button to indicate that the continuous shooting mode is activated, and to call the multi-frame fusion algorithm pre-built in the algorithm library to implement the image processing method provided in this application.

[0211] The phone manager is used to provide communication functions for the mobile phone 100. For example, it manages call status (including connection, hang-up, etc.).

[0212] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0213] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0214] Android TM Runtime includes core libraries and a virtual machine. Android TMruntime is responsible for Android TM System scheduling and management.

[0215] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0216] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0217] The system library may include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (e.g., OpenGL ES), a 2D graphics engine (e.g., SGL), and an algorithm library. The algorithm library may pre-load or pre-configure multi-frame fusion algorithms for implementing the image processing methods provided in this application, for use by the aforementioned camera service.

[0218] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0219] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0220] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0221] A 2D graphics engine is a graphics engine for 2D drawing.

[0222] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0223] The following example illustrates the workflow of the mobile phone's software and hardware in the context of capturing and photographing scenes.

[0224] When touch sensor 180K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer and identifies the control corresponding to the input event. Taking a touch click as an example, where the corresponding control is the camera application icon, the camera application calls the application framework layer's interface to launch the camera application, and then calls the kernel layer to launch the camera driver, capturing still images or videos through camera 193.

[0225] This application also provides a computer program product for implementing the image processing methods provided in the above embodiments. The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer program modules or module code executable on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0226] Computer program modules or module code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0227] Module code can be implemented using a high-level modular language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used to implement module code when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0228] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, optical discs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other forms of propagated signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0229] In this specification, the reference to "an embodiment" or "an embodiment" means that a specific feature, structure, or characteristic described in connection with the embodiment is included in at least one exemplary implementation or technology disclosed according to an embodiment of this application. The appearance of the phrase "in an embodiment" in various places in the specification does not necessarily refer to the same embodiment.

[0230] The disclosure of embodiments of this application also relates to means for performing operations in text. This means may be specifically constructed for the claimed purpose or may include a general-purpose computer selectively activated or reconfigured by a computer program stored in a computer. Such a computer program may be stored on a computer-readable medium, such as, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory, random access memory, EPROMs, EEPROMs, magnetic or optical cards, application-specific integrated circuits (ASICs), or any type of medium suitable for storing electronic instructions, and each may be coupled to a computer system bus. Furthermore, the computer mentioned in the specification may include a single processor or may employ an architecture involving multiple processors for increased computing power.

[0231] Furthermore, the language used in this specification has been primarily chosen for readability and instructional purposes and may not have been chosen to depict or limit the disclosed subject matter. Therefore, the embodiments disclosed herein are intended to illustrate, and not limit, the scope of the concepts discussed herein.

Claims

1. An image processing method applied to an electronic device, comprising: The method comprises: ​ detecting a continuous shooting instruction; in response to the continuous shooting instruction, collecting M frames of images, wherein the continuous shooting instruction triggers N shooting events, and M>N≥2; determining N frames of reference frames and noise reduction frames corresponding to each of the reference frames from the M frames of images, wherein each of the reference frames corresponds to K frames of noise reduction frames, and K≥2; performing fusion noise reduction on each of the reference frames based on the noise reduction frames corresponding to each of the reference frames, to obtain N frames of event images corresponding to the N shooting events.

2. The method of claim 1, wherein, The determining of the N frames of reference frames and the noise reduction frames corresponding to each of the reference frames from the M frames of images comprises: sorting the M frames of images according to shooting times to obtain a first image sequence; determining the N frames of reference frames from the first image sequence; based on the N frames of reference frames, determining P noise reduction groups, wherein the P noise reduction groups comprise a first noise reduction group, the first noise reduction group comprises two adjacent first and second reference frames and L frames of images between the first and second reference frames, 1≤L≤(K-1), and the noise reduction frames corresponding to the first and second reference frames both comprise the L frames of images.

3. The method of claim 2, wherein, The noise reduction frame of the first reference frame further comprises the second reference frame, and the noise reduction frame of the second reference frame further comprises the first reference frame.

4. The method according to claim 2 or 3, characterized in that, When N is an odd number, the P noise reduction groups further comprise a second noise reduction group, the second noise reduction group comprises a third reference frame and K frames of noise reduction frames.

5. The method of claim 4, wherein, The third reference frame is located after an (N-1)th reference frame in the N frames of reference frames in the first image sequence.

6. The method of claim 5, wherein, Each of the reference frames is a first frame of image in n frames of images collected for each of the shooting events, wherein M equals N*n.

7. The method of claim 6, wherein, The interval L frames of images between the two adjacent reference frames satisfy that L equals n-2 or L equals n-1.

8. The method of claim 2, wherein, The performing of the fusion noise reduction on each of the reference frames based on the noise reduction frames corresponding to each of the reference frames comprises: calculating and saving a first registration matrix of the second reference frame and the L frames of images relative to the first reference frame; performing fusion noise reduction on the first reference frame based on the second reference frame, image data of the L frames of images and the first registration matrix; and performing inverse matrix conversion based on the first registration matrix to obtain a second registration matrix of the first reference frame and the L frames of images relative to the second reference frame; performing fusion noise reduction on the second reference frame based on the first reference frame, image data of the L frames of images and the second registration matrix.

9. The method of claim 1, wherein, The determining of the N frames of reference frames and the noise reduction frames corresponding to each of the reference frames from the M frames of images further comprises: sorting the M frames of images according to shooting times to obtain a second image sequence; determining the N frames of reference frames from the second image sequence, wherein each of the reference frames is a first frame of image in n frames of images collected for each of the shooting events. Based on the N reference frames, Q noise reduction groups are determined, wherein the Q noise reduction groups include a third noise reduction group and a fourth noise reduction group adjacent to each other, the third noise reduction group includes two adjacent fourth and fifth reference frames and L frames of images between the fourth and fifth reference frames, the fourth noise reduction group includes two adjacent sixth and seventh reference frames and L frames of images between the sixth and seventh reference frames, and R frames of images between the fifth reference frame and the sixth reference frame satisfy R not equal to L.

10. An electronic device, comprising: The electronic device comprises: one or more processors; one or more memories, the one or more memories storing one or more programs, when the one or more programs are executed by the one or more processors, causing the electronic device to perform the image processing method in any one of claims 1 to 9.

11. A computer readable medium characterized by The readable medium stores instructions, when the instructions are executed on a computer, causing the computer to perform the image processing method in any one of claims 1 to 9.

12. A computer program product, characterised in that, The computer program / instructions, when executed by a processor, implement the image processing method in any one of claims 1 to 9.

Images