Photographing method, electronic device, storage medium and chip system

Abstract

The application is suitable for the terminal technical field, and provides a shooting method, an electronic device, a storage medium and a chip system. The method comprises the following steps: in the case that a shooting instruction is received and a to-be-shot scene in a shooting preview frame is a target scene, shooting at least one group of images; the target scene comprises a fireworks scene; each group of images comprises multiple images with the same exposure time and different exposure amounts which are shot at the same time; an image with a smaller exposure amount in each group of images is shot by a camera with better photosensitive performance, and an image with a larger exposure amount is shot by a camera with poorer photosensitive performance; the shooting time of different groups of images is different; performing first processing on each group of images, so that the field of view, size and resolution of all images in each group of images are all the same; inputting multiple to-be-fused image groups with adjacent shooting time into a trained image denoising fusion model for processing to obtain a fused image, so that the overall quality of a shot fireworks image can be improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a shooting method, electronic device, storage medium and chip system. Background Technology

[0002] In daily life, people sometimes use mobile phones and other electronic devices to take pictures of fireworks. Because the process of fireworks exploding is fleeting, and the contrast between the bright sparks produced by the fireworks and the dark night sky is quite obvious, in order to preserve the details of both the brightest and darkest parts of the fireworks image and improve the overall quality of the image, electronic devices usually use a single-camera multi-frame multi-exposure fusion shooting strategy to capture fireworks images. That is, multiple frames of fireworks images are captured sequentially by a single camera, the multiple frames of fireworks images are merged, and the merged fireworks image is output.

[0003] However, due to insufficient light intake by the image sensor in a nighttime camera, each frame of a fireworks image requires a relatively long exposure time to ensure adequate exposure for each frame captured by a single camera. Since the sparks produced after a firework explode move quickly, a long exposure time results in significant differences in the spark content between adjacent frames. Consequently, fusing multiple frames with such significant spark content discrepancies produces a merged fireworks image with large areas of gaps, leading to poor overall quality in the final output image. Summary of the Invention

[0004] This application provides a shooting method, electronic device, storage medium, and chip system that can improve the overall quality of fireworks images captured by the electronic device.

[0005] In a first aspect, embodiments of this application provide a shooting method, comprising: upon receiving a shooting instruction and when the scene to be shot in the shooting preview frame is a target scene, capturing at least one set of images using multiple cameras; the target scene includes a fireworks scene; each set of images includes: multiple images of the scene to be shot simultaneously captured by multiple cameras with the same exposure time but different exposure amounts; in each set of images, the image with a smaller exposure amount is captured by the camera with better light sensitivity among the multiple cameras, and the image with a larger exposure amount is captured by the camera with poorer light sensitivity among the multiple cameras; the shooting times of different sets of images are different; performing a first processing on each set of images to make all images in each set of images have the same field of view, the same size, and the same resolution; each set of images after the first processing is a set of images to be fused; inputting multiple sets of images to be fused with adjacent shooting times into a trained image denoising and fusion model for processing to obtain a fused image.

[0006] The electronic device receiving the shooting instruction may include, but is not limited to:

[0007] The electronic device detects clicks or double-clicks on the shooting controls in the shooting preview interface;

[0008] Alternatively, the electronic device detects a quick shooting operation while displaying a shooting preview interface;

[0009] Alternatively, the electronic device may receive a voice command to take a picture while displaying a shooting preview interface;

[0010] Alternatively, the electronic device may receive a shooting command from another electronic device while displaying a shooting preview interface.

[0011] Optionally, the electronic device can perform target recognition on the scene to be shot in the shooting preview frame in real time after displaying the shooting preview interface, so as to determine whether the scene to be shot is the target scene.

[0012] Optionally, after receiving the shooting instruction, the electronic device can perform target recognition on the scene to be shot in the shooting preview frame to determine whether the scene to be shot is the target scene.

[0013] For example, the target scene can refer to a high-brightness scene that changes dynamically at night. For instance, the target scene may include a fireworks scene or a scene of light sources moving rapidly at night (such as car headlights or scrolling signs).

[0014] It's important to note that "good light sensitivity" and "poor light sensitivity" are relative terms. Specifically, good light sensitivity means the image sensor in the camera is generally more sensitive to light. Poor light sensitivity means the image sensor in the camera is generally less sensitive to light. For example, in a multi-camera setup including a main camera and a telephoto camera, since the image sensor in the main camera is generally more sensitive to light than the image sensor in the telephoto camera, the main camera is considered to have good light sensitivity, while the telephoto camera is considered to have poor light sensitivity. Similarly, low exposure and high exposure are also relative terms. For instance, assuming each set of images includes a first exposure image and a second exposure image, and the first exposure image has a lower exposure than the second exposure image, then the first exposure image in each set is considered to have lower exposure, and the second exposure image is considered to have higher exposure.

[0015] In practical applications, the field of view of multiple different cameras used to capture each set of images can be all the same, partially the same, or all different. In some embodiments, the multiple different cameras may include a main camera and a telephoto camera. In other embodiments, the multiple different cameras may include a main camera and a wide-angle camera. In still other embodiments, the multiple different cameras may include two main cameras with different performance characteristics. In yet another embodiment, the multiple different cameras may include a telephoto camera, a main camera, and a wide-angle camera.

[0016] Since the exposure of an image is determined by the aperture, exposure time, and ISO of the camera that captured the image, electronic devices can control the exposure of each image in each group to be different, while ensuring that the exposure time of multiple images in each group is the same:

[0017] For electronic devices with a fixed aperture value and an adjustable ISO value, the electronic device can control the ISO value of a camera with better light sensitivity to be lower than that of a camera with poorer light sensitivity.

[0018] For electronic devices with adjustable aperture values ​​and fixed ISO values, the electronic device can control the aperture value of a camera with better light sensitivity to be greater than that of a camera with poorer light sensitivity.

[0019] For electronic devices where both the aperture and ISO values ​​of the camera are adjustable, the electronic device can control the ISO value of the camera with better light sensitivity to be lower than that of the camera with poorer light sensitivity, and / or control the aperture value of the camera with better light sensitivity to be greater than that of the camera with poorer light sensitivity.

[0020] The field of view of the image can refer to the field of view of the camera that captured the image. The camera's field of view can also refer to the maximum range of angles that the camera can capture.

[0021] Image size can refer to the actual size of the image in physical space.

[0022] Image resolution can be used to represent the distribution of pixels in the width and height of an image.

[0023] According to the shooting method provided in this application, since the same camera is only used to capture images under the same exposure level, rather than to capture images under multiple different exposure levels, the continuity of multiple frames of images under the same exposure level captured by the same camera can be improved, and the inter-frame differences of multiple frames of images under the same exposure level can be reduced, thereby reducing the registration difficulty during subsequent image registration. Simultaneously, since images under different exposure levels are captured by different cameras, and the shooting time and exposure duration of images under different exposure levels are the same, it can be ensured that the content of images under different exposure levels is highly consistent, thereby further reducing the registration difficulty during subsequent image registration and improving the quality of the final fused image. Furthermore, since images under different exposure levels are captured simultaneously by multiple cameras, rather than by a single camera, the shooting efficiency of the fused image can be improved, the user's shooting waiting time can be shortened, and the user's shooting experience can be enhanced.

[0024] Furthermore, in this embodiment, an image with a smaller exposure is captured by a camera with better light sensitivity, while an image with a larger exposure is captured by a camera with poorer light sensitivity. This can simultaneously improve the signal-to-noise ratio of both the image with a smaller exposure and the image with a larger exposure. Consequently, the electronic device can then fuse the image with the smaller exposure and the image with the larger exposure to create a higher-quality fused image, thereby further improving the overall quality of the fireworks image captured at night.

[0025] In one optional implementation of the first aspect, the image denoising fusion model includes multiple denoising units and an image fusion unit; the total number of multiple denoising units is equal to the total number of multiple cameras, and each denoising unit corresponds one-to-one with a multiple camera; the denoising unit is used to fuse and denoise multiple images with the same exposure taken by the corresponding camera; correspondingly, multiple groups of images to be fused that are adjacent in shooting time are input into the trained image denoising fusion model for processing to obtain a fused image, including: through each denoising unit, fusing and denoising multiple images with the same exposure taken by the camera corresponding to the denoising unit, and outputting a denoised image to the image fusion unit respectively; through the image fusion unit, fusing multiple denoised images with different exposures from multiple denoising units to obtain a fused image.

[0026] According to the shooting method provided in the embodiments of this application, since the image denoising fusion model is configured with multiple denoising units that correspond one-to-one with multiple cameras, and different denoising units are used to fuse and denoise multiple frames of images with the same exposure taken by different cameras, the image denoising fusion model can achieve accurate denoising of images with different exposures, which is beneficial to improving the overall quality of the fused image output by the image denoising fusion model.

[0027] In one optional implementation of the first aspect, different denoising units are trained using different training datasets. The training datasets include multiple sample data sets, each containing multiple noisy images and one real image without noise. The noisy images are used as input to the denoising units during training, and the real image is used as output. Correspondingly, the training datasets for each of the multiple denoising units are generated as follows: acquiring multiple high-definition night scene images; for each high-definition night scene image, randomly cropping a region from the high-definition night scene image as the image to be degraded; performing random preprocessing on the image to be degraded; the random preprocessing includes: random white balance processing, random rotation, random addition of ghosting, random addition of color blocks, and random addition of jitter blur; and then processing the preprocessed image to be degraded... The image is identified as a real image; multiple copies of the real image are obtained; the total number of copies is equal to the total number of cameras; each copy corresponds to a different camera among the multiple cameras; a first noise-adding process is applied to the copy image corresponding to the large field-of-view camera among the multiple cameras to obtain a noisy image corresponding to the large field-of-view camera; a second noise-adding process is applied to the copy image corresponding to the small field-of-view camera among the multiple cameras to obtain a noisy image corresponding to the small field-of-view camera; the second noise-adding process is different from the first noise-adding process; the noisy image corresponding to the large field-of-view camera and the real image are used as a sample data for a denoising unit corresponding to the large field-of-view camera; the noisy image corresponding to the small field-of-view camera and the real image are used as a sample data for a denoising unit corresponding to the small field-of-view camera.

[0028] The high-definition night scene image can be any night scene image captured by professional shooting equipment (such as an SLR camera or a point-and-shoot camera). For example, the high-definition night scene image can be an image in RGB format.

[0029] In one optional implementation of the first aspect, a first noise-adding process is performed on the copy image corresponding to the wide field-of-view camera among multiple cameras to obtain a noisy image corresponding to the wide field-of-view camera, including: performing image quality blurring processing on the copy image corresponding to the wide field-of-view camera; performing photosensitivity compensation on the copy image after image quality blurring processing; adding first noise to the photosensitivity-compensated copy image based on the noise coefficient of the wide field-of-view camera; and converting the copy image with the added first noise into a raw format image to obtain the noisy image corresponding to the wide field-of-view camera.

[0030] Among them, raw format can be understood as raw format.

[0031] For example, the first noise may include a first Gaussian white noise and a first Poisson particle noise corresponding to the noise figure of the wide field-of-view camera.

[0032] For example, an electronic device can perform a random downsampling operation on a copy image corresponding to a wide field-of-view camera, and then perform a random upsampling operation on the copy image after the random downsampling operation, in order to add image blur caused by image cropping to the copy image corresponding to the wide field-of-view camera.

[0033] Electronic devices can multiply the pixel value of each pixel in the copy image corresponding to the wide field-of-view camera, which has undergone image quality blurring, by a sensitivity difference value to achieve sensitivity compensation for the copy image corresponding to the wide field-of-view camera.

[0034] In one optional implementation of the first aspect, a second noise-adding process is performed on the copy image corresponding to the small field-of-view camera among multiple cameras to obtain a noisy image corresponding to the small field-of-view camera, including: brightening the copy image corresponding to the small field-of-view camera; adding second noise to the brightened copy image based on the noise coefficient of the small field-of-view camera; and converting the copy image with added second noise into a raw format image to obtain the noisy image corresponding to the small field-of-view camera.

[0035] For example, the second noise may include a second Gaussian white noise and a second Poisson particle noise corresponding to the noise figure of the small field-of-view camera.

[0036] Since electronic devices can obtain one sample data point corresponding to each denoising unit from each high-definition night scene image, the training dataset corresponding to each denoising unit can be obtained from multiple high-definition night scene images.

[0037] According to the shooting method provided in the embodiments of this application, since different denoising units in the image denoising fusion model are trained by training datasets that match their required denoising capabilities, different denoising units can have different denoising capabilities, thereby enabling the image denoising fusion model to perform more accurate denoising on images taken by different cameras under different exposure levels, thereby improving the overall quality of the fused image output by the image denoising fusion model.

[0038] In one optional implementation of the first aspect, the first processing includes: cropping, scaling, and upsampling; correspondingly, the first processing is performed on each group of images, including: cropping the large field-of-view image in each group of images to obtain a cropped image corresponding to the large field-of-view image; the field of view corresponding to the cropped image is the same as the field of view corresponding to the small field-of-view image in the same group of images; the large field-of-view image refers to an image captured by a large field-of-view camera among multiple cameras, and the small field-of-view image refers to an image captured by a small field-of-view camera among multiple cameras; scaling the cropped image corresponding to the large field-of-view image in each group of images. Obtain the scaled image corresponding to the large field-of-view image; the size of the scaled image is the same as the size of the small field-of-view image in the same group of images, and the field of view corresponding to the scaled image is the same as the field of view corresponding to the small field-of-view image in the same group of images; upsample the scaled image corresponding to the large field-of-view image in each group of images to obtain the upsampled image corresponding to the large field-of-view image; the resolution of the upsampled image is the same as the resolution of the small field-of-view image in the same group of images, and the size of the upsampled image is the same as the size of the small field-of-view image in the same group of images, and the field of view corresponding to the upsampled image is the same as the field of view corresponding to the small field-of-view image in the same group of images.

[0039] It should be noted that the terms "wide field of view image" and "narrow field of view image" are relative. Specifically, a wide field of view image refers to an image captured by the camera with the largest field of view among multiple cameras, while a narrow field of view image refers to an image captured by the camera with the smallest field of view among multiple cameras. For example, suppose that the first exposure image in each set of images is captured by the main camera and the second exposure image is captured by the telephoto camera. Since the field of view of the main camera is larger than that of the telephoto camera, the first exposure image in each set of images is a wide field of view image, and the second exposure image is a narrow field of view image.

[0040] According to the shooting method provided in the embodiments of this application, by processing the large field-of-view image in each group of images to keep the field of view, size and resolution consistent with the small field-of-view image, the registration accuracy during subsequent image registration can be improved.

[0041] In one optional implementation of the first aspect, the first processing further includes: distortion correction; correspondingly, performing the first processing on each group of images further includes: performing distortion correction on each image based on the calibration intrinsic parameters of the camera corresponding to each image in each group of images; the calibration intrinsic parameters include distortion coefficients and focal length.

[0042] Among them, the calibration parameters of a camera can refer to the pre-calibrated internal parameters of the camera.

[0043] For example, the internal parameters of a camera may include its distortion coefficients and focal length. The distortion coefficients may include radial distortion coefficients and tangential distortion coefficients. The radial distortion coefficient describes the degree to which individual pixels in an image are bent outwards or inwards as they move further from the center pixel. The tangential distortion coefficient describes the degree of skewness of individual pixels in the image caused by the camera not being aligned with the image plane. The focal length of the camera may include its horizontal focal length and its vertical focal length.

[0044] According to the shooting method provided in the embodiments of this application, by performing distortion correction on each image in each group of images, the images captured by different cameras can be corrected to the same reference plane, thereby reducing or eliminating the distortion differences between the images in each group of images and improving the registration accuracy during subsequent image registration.

[0045] In one optional implementation of the first aspect, the first processing further includes: affine transformation; correspondingly, performing the first processing on each group of images further includes: performing affine transformation on each image based on the calibration extrinsic parameters of the camera corresponding to each image in each group of images; the calibration extrinsic parameters are used to describe the pose of the camera relative to the spatial coordinate system.

[0046] The calibration extrinsic parameters of a camera refer to the pre-calibrated external parameters of the camera. These external parameters can be used to describe the camera's pose, for example, its pose relative to the world coordinate system.

[0047] For example, the external parameters of the camera may include: the translation of the camera in the three degrees of freedom of the spatial coordinate system (i.e., the x-axis, y-axis and z-axis), and the rotation angle of the camera around the three degrees of freedom, etc.

[0048] According to the shooting method provided in the embodiments of this application, by performing an affine transformation on each image in each group of images, all images in each group of images can be aligned to a unified reference coordinate system, thereby reducing or eliminating the physical viewing angle difference between each image in each group of images and improving the registration accuracy during subsequent image registration.

[0049] In one alternative implementation of the first aspect, the first processing may further include: inter-group registration. Correspondingly, performing the first processing on each group of images further includes: performing inter-group registration on multiple images in each group.

[0050] For example, an electronic device can use the image with lower exposure in each group of images as a reference frame, and align the image with higher exposure in the same group to the image with lower exposure, thereby achieving registration between multiple images in the same group. Aligning the image with higher exposure to the image with lower exposure can include aligning each pixel in the image with a corresponding pixel (i.e., a pixel with identical content) in the image with lower exposure.

[0051] According to the shooting method provided in the embodiments of this application, by performing inter-group registration on each group of images, the content of images taken by different cameras at different exposure levels can be aligned as much as possible, thereby improving the subsequent image fusion effect.

[0052] In one alternative implementation of the first aspect, the first processing may further include: brightness alignment. Correspondingly, performing the first processing on each group of images further includes: calculating the difference in exposure values ​​between every two images in each group of images, and adjusting the brightness of each pair of images based on the exposure difference between each pair of images, so that the brightness of all images in each group of images is the same.

[0053] According to the shooting method provided in the embodiments of this application, by performing brightness alignment on each image in each group of images, the brightness of all images in each group of images can be made consistent, which is beneficial to improving the subsequent image fusion effect.

[0054] In one alternative implementation of the first aspect, after performing the first processing on each set of images, the method further includes:

[0055] Perform inter-frame registration on all images in all groups of images.

[0056] According to the shooting method provided in the embodiments of this application, by registering all images in all groups of images, all images can be further aligned, thereby further improving the subsequent image fusion effect.

[0057] Secondly, embodiments of this application provide an electronic device, including: one or more processors, and a memory;

[0058] The memory is coupled to one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and the one or more processors calling the computer instructions to cause the electronic device to perform the imaging method as described in any of the implementations of the first aspect above.

[0059] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform a shooting method as described in any of the implementations of the first aspect above.

[0060] Fourthly, embodiments of this application provide a computer-executable program product that, when run on an electronic device, causes the electronic device to execute the shooting method of any implementation of the first aspect described above.

[0061] Fifthly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to execute a shooting method as described in any of the implementations of the first aspect above.

[0062] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0063] Figure 1 This is a schematic diagram showing the field of view of a main camera, a wide-angle camera, and a telephoto camera, respectively.

[0064] Figure 2 This is a schematic diagram of a shooting strategy based on multi-frame multi-exposure fusion of a single camera to capture a fireworks scene;

[0065] Figure 3 This is a schematic diagram of a fused fireworks image obtained using a single-camera multi-frame multi-exposure fusion shooting strategy;

[0066] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0067] Figure 5 A schematic architecture diagram of a software system for an electronic device provided in an embodiment of this application;

[0068] Figure 6 A schematic flowchart illustrating a shooting method provided in an embodiment of this application;

[0069] Figure 7 A schematic diagram of a shooting preview interface provided in an embodiment of this application;

[0070] Figure 8 This is a schematic diagram of multiple images captured by an electronic device based on the shooting method provided in the embodiments of this application;

[0071] Figure 9 This is a schematic diagram illustrating the specific implementation process of S603 in a shooting method provided in an embodiment of this application;

[0072] Figure 10A schematic diagram illustrating the processing steps involved in performing a first processing on each group of images by an electronic device provided in an embodiment of this application;

[0073] Figure 11 This is a schematic diagram illustrating the specific implementation process of S603 in a shooting method provided in another embodiment of this application;

[0074] Figure 12 This is a schematic diagram of the structure of an image denoising and fusion model provided in an embodiment of this application;

[0075] Figure 13 This is a schematic diagram illustrating the process of generating the training dataset corresponding to the denoising unit in an image denoising fusion model provided in this application embodiment. Detailed Implementation

[0076] It should be noted that the terminology used in the implementation section of the embodiments of this application is only used to explain the specific embodiments of this application and is not intended to limit this application. In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related items, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. In addition, in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, "at least one" or "one or more" means one, two or more.

[0077] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0078] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0079] First, some of the terms used in the embodiments of this application will be explained so that those skilled in the art can understand them.

[0080] 1. Aperture is a component inside a camera used to control the amount of light entering the image sensor. The size of the aperture is expressed by the aperture value. Specifically, the smaller the aperture value, the larger the aperture, and the more light enters the image sensor; the larger the aperture value, the smaller the aperture, and the less light enters the image sensor.

[0081] 2. Shutter speed, used to indicate the length of time the camera's shutter is open to allow light to enter the image sensor (i.e., exposure time). The faster the shutter speed, the shorter the exposure time; the slower the shutter speed, the longer the exposure time.

[0082] 3. Sensitivity, used to indicate the light sensitivity of the image sensor in a camera. Sensitivity is expressed by a sensitivity value. For example, sensitivity values ​​can be defined by the International Organization for Standardization (ISO), which can be simply referred to as ISO values. Specifically, a smaller ISO value indicates lower light sensitivity of the image sensor, making it more suitable for well-lit shooting scenarios; a larger ISO value indicates higher light sensitivity of the image sensor, making it more suitable for low-light shooting scenarios.

[0083] 4. Exposure: This refers to the amount of light actually received by the image sensor in the camera, which directly affects the brightness of the image. Specifically, with moderate exposure, the image has a balanced brightness; with excessive exposure (overexposure), the image is usually too bright; and with insufficient exposure (underexposure), the image is usually too dark.

[0084] Exposure can be determined by aperture value, shutter speed, and ISO value. For example, the amount of exposure can be represented by the exposure value (EV). Specifically, a larger EV value indicates a smaller exposure; a smaller EV value indicates a larger exposure. The relationship between EV value, aperture value, shutter speed, and ISO value can be expressed by the following formula (1):

[0085]

[0086] Where EV represents the exposure value, N represents the aperture value, t represents the exposure time, and ISO represents the ISO value.

[0087] According to formula (1) above, when the exposure time and ISO value remain constant, a larger aperture value (i.e., a smaller aperture) results in a larger exposure value and a smaller exposure amount; conversely, a smaller aperture value (i.e., a larger aperture) results in a smaller exposure value and a larger exposure amount. When both aperture and ISO values ​​remain constant, a shorter exposure time results in a larger exposure value and a smaller exposure amount; conversely, a longer exposure time results in a smaller exposure value and a larger exposure amount. When both aperture and exposure time remain constant, a smaller ISO value results in a larger exposure value and a smaller exposure amount; conversely, a larger ISO value results in a smaller exposure value and a larger exposure amount.

[0088] 5. Field of View (FOV): This indicates the maximum angular range that a camera can capture. When the object to be photographed is within this angular range, the object can be captured by the camera; when the object is outside this angular range, the object will not be captured by the camera. Generally, the larger the FOV of a camera, the larger the shooting range and the shorter the focal length; the smaller the FOV, the smaller the shooting range and the longer the focal length. Therefore, based on different FOVs, cameras can be classified as main cameras, wide-angle cameras, and telephoto cameras, etc.

[0089] For example, please refer to Figure 1 This diagram illustrates the field of view of a main camera, a wide-angle camera, and a telephoto camera. It shows that the wide-angle camera has a larger field of view than the main camera, making it more suitable for close-up shots. The telephoto camera has a smaller field of view than the main camera, making it more suitable for distant shots. The main camera is best suited for shooting scenes at a moderate distance.

[0090] The above is a brief introduction to some of the terms used in the embodiments of this application, and will not be repeated below.

[0091] In daily life, people sometimes celebrate festivals by setting off fireworks and use mobile phones and other electronic devices to capture the beautiful moments of fireworks blooming in the night sky. Because the blooming of fireworks is fleeting, and the contrast between the bright sparks and the dark night sky is quite pronounced, electronic devices typically employ a single-camera, multi-frame, multi-exposure fusion shooting strategy when capturing dynamic, high-brightness scenes like fireworks at night. This strategy aims to preserve details of both the brightest and darkest parts of the fireworks image, thereby expanding its dynamic range and improving its overall quality.

[0092] The single-camera multi-frame multi-exposure fusion shooting strategy is as follows: For the same scene to be shot, multiple frames of scene images under different exposure levels are captured sequentially using the same camera. These sequentially captured images under different exposure levels are then registered and fused, and a fused image is output. The different exposure levels are achieved by controlling the exposure time differently while keeping other parameters affecting the exposure (such as aperture and ISO values) constant. For an example, please refer to [link to example]. Figure 2 This is a schematic diagram illustrating a shooting strategy based on multi-frame, multi-exposure fusion using a single camera to capture a fireworks scene. Figure 2 As shown, when an electronic device captures images of fireworks, it can sequentially capture images of the fireworks scene at a first exposure level and a second exposure level using a telephoto camera. These multiple frames of fireworks images captured sequentially at the first and second exposure levels are then registered and fused to obtain a fused fireworks image. The first exposure level is greater than the second exposure level; that is, the first exposure level is a larger exposure level, and the second exposure level is a smaller exposure level. The specific values ​​of the first and second exposure levels can be set according to actual conditions.

[0093] However, when electronic devices capture fireworks images at night using the aforementioned shooting strategy, the image sensor in the camera receives insufficient light at night. Therefore, to ensure that the exposure of each frame of the fireworks image meets the requirements, a relatively long exposure time is required for each frame. Since the sparks produced after the fireworks bloom move quickly, a longer exposure time leads to significant differences in the spark content between adjacent frames (i.e., large inter-frame differences). Thus, fusing multiple sequentially captured fireworks images with large inter-frame differences at different exposures results in a merged fireworks image with large areas of fusion holes and poor high dynamic range (HDR) performance, leading to a poor overall quality of the final output merged fireworks image.

[0094] For example, please refer to Figure 3 This is a schematic diagram of a fused fireworks image obtained using a single-camera, multi-frame, multi-exposure fusion shooting strategy. It should be noted that... Figure 3 The fireworks images at the first and second exposures are two consecutive frames taken sequentially by the same camera of the same fireworks scene. Figure 3It can be seen that the spark content of the fireworks image under the first exposure level differs significantly from that under the second exposure level. Specifically, taking the overexposed area 30 in the fireworks image under the first exposure level as an example, the corresponding area in the fireworks image under the second exposure level should be bright, but the corresponding area in the actual second exposure image is dark. This causes the overexposed area 30 in the fireworks image under the first exposure level to fail to align with the bright area in the fireworks image under the second exposure level. In other words, it is impossible to find a bright area in the fireworks image under the second exposure level that can be registered with the overexposed area 30 to enrich the details of the overexposed area 30, resulting in an overexposure anomaly resembling a hole (i.e., a fusion hole) in the merged fireworks image. In addition, in some scenarios, false color problems may also occur in the merged fireworks image. For example, as shown in... Figure 3 As shown, the false color problem refers to the fact that the color of the sparks in the merged fireworks image changes compared to the color of the sparks in the original image, resulting in the color of the sparks in the merged fireworks image being inconsistent with the color of the sparks in the actual scene.

[0095] Therefore, related technologies typically address issues like fusion holes and false colors in merged fireworks images by reducing the exposure time of the fireworks images. This involves using the shortest possible exposure time to ensure consistency between adjacent frames. However, because the sparks from fireworks move rapidly after they explode, simply reducing the exposure time cannot fundamentally solve the problem of significant differences between frames. Furthermore, shorter exposure times result in lower signal-to-noise ratios in the captured fireworks images, leading to poorer overall quality of the merged fireworks image.

[0096] In view of this, embodiments of this application provide a shooting method, including: upon receiving a shooting instruction and when the scene to be shot in the shooting preview frame is a target scene, capturing at least one set of images using multiple cameras; the target scene includes a fireworks scene; each set of images includes: multiple images of the scene to be shot simultaneously captured by multiple cameras with the same exposure time but different exposure amounts; in each set of images, the image with a smaller exposure amount is captured by the camera with better light sensitivity among the multiple cameras, and the image with a larger exposure amount is captured by the camera with poorer light sensitivity among the multiple cameras; the shooting times of different sets of images are different; performing a first processing on each set of images to make all images in each set of images have the same field of view, the same size, and the same resolution; each set of images after the first processing is a set of images to be fused; inputting multiple sets of images to be fused with adjacent shooting times into a trained image denoising and fusion model for processing to obtain a fused image.

[0097] In this embodiment, since the same camera is used only to capture images at the same exposure level, rather than capturing images at multiple different exposure levels, the continuity of multiple frames captured by the same camera at the same exposure level can be improved, reducing the inter-frame differences between multiple frames at the same exposure level, thereby reducing the registration difficulty during subsequent image registration. Simultaneously, since images at different exposure levels are captured by different cameras, and the capture time and exposure duration are all the same, it can be ensured that the content of images at different exposure levels is highly consistent, further reducing the registration difficulty during subsequent image registration and improving the quality of the final fused image. Furthermore, since images at different exposure levels are captured simultaneously by multiple cameras, rather than by a single camera, the capture efficiency of the fused image can be improved, shortening the user's shooting waiting time and enhancing the user's shooting experience.

[0098] Furthermore, in this embodiment, an image with a smaller exposure is captured by a camera with better light sensitivity, while an image with a larger exposure is captured by a camera with poorer light sensitivity. This can simultaneously improve the signal-to-noise ratio of both the image with a smaller exposure and the image with a larger exposure. Consequently, the electronic device can then fuse the image with the smaller exposure and the image with the larger exposure to create a higher-quality fused image, thereby further improving the overall quality of the fireworks image captured at night.

[0099] The shooting method provided in this application can be applied to various electronic devices. Exemplarily, electronic devices may include SLR cameras, point-and-shoot cameras and other video recording devices, mobile phones, tablets, wearable devices, augmented reality (AR) devices, virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks and personal digital assistants (PDAs), etc. This application does not limit the specific type of electronic device.

[0100] The following uses a mobile phone as an example to illustrate the structure of an electronic device.

[0101] Please see Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Figure 4As shown, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180I, a touch sensor 180J, a bone conduction sensor 180K, an ambient light sensor 180L, etc.

[0102] Processor 110 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0103] In practical applications, electronic devices can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194, and application processor.

[0104] The camera 193 can be used to capture images. Exemplarily, the camera 193 may include components such as a lens, a filter, and an image sensor. When the camera 193 captures an image, light emitted or reflected by an object enters the lens of the camera 193, passes through the filter, and is finally converged onto the image sensor. The image sensor is mainly used to converge and image the light emitted or reflected by all objects (also referred to as the scene to be captured) within the field of view of the camera 193. The filter is mainly used to filter out excess light waves (such as light waves other than visible light, such as infrared light waves) from the light.

[0105] Specifically, the image sensor can be used to convert received light into electrical signals and transmit these signals to the image sensor (ISP). The ISP can be used to convert the electrical signals into digital image signals and transmit these signals to the digital image DSP. The DSP can be used to convert the digital image signals into image signals in standard primary color (RGB) or YUV formats. Here, Y represents luminance, and U and V represent chrominance. In some embodiments, the ISP can be located in the camera 193.

[0106] For example, an image sensor may be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor, etc.

[0107] In some embodiments, the camera 193 can be disposed on the front or the back of the electronic device. This application does not impose any particular limitation on the specific number or arrangement of the cameras 193.

[0108] For example, an electronic device may include a front-facing camera and a rear-facing camera. Both the front-facing and rear-facing cameras may include one or more cameras. Taking an electronic device with three rear-facing cameras—a main camera, a telephoto camera, and a wide-angle camera—as an example, the electronic device can employ the shooting method provided in this application embodiment when simultaneously activating at least two rear-facing cameras or simultaneously activating at least two front-facing cameras for shooting.

[0109] Understandable, Figure 4 The illustrated structure does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine or separate certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0110] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture, etc. This application uses the layered architecture Android system as an example to illustrate the software system of an electronic device.

[0111] Please see Figure 5 This is a schematic architecture diagram of a software system for an electronic device provided in an embodiment of this application.

[0112] like Figure 5 As shown, a layered architecture can divide software into several layers, each with a clear role and division of labor, and the layers communicate with each other through software interfaces. For example, a layered architecture can divide the Android system into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0113] The application layer can include a series of application packages. Application packages may include, for example, applications for camera, gallery, calendar, calling, maps, navigation, WLAN, Bluetooth, music, video, and SMS.

[0114] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The application framework layer may include some predefined functions.

[0115] For example, the application framework layer may include a window manager, a content provider, a phone manager, a resource manager, a notification manager, a view system, etc.

[0116] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0117] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0118] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0119] System libraries can include multiple functional modules. For example: surface manager, 3D graphics processing library (e.g., OpenGL ES), 2D graphics engine (e.g., SGL), media libraries, etc.

[0120] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, and sensor drivers.

[0121] It should be noted that, Figure 5 Only the modules relevant to the embodiments of this application are shown. In other embodiments, each layer may include any other possible modules, and each module may include one or more sub-modules. This application does not limit these modules.

[0122] The imaging method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0123] Please see Figure 6 This is a schematic flowchart illustrating a shooting method provided in an embodiment of this application. This shooting method can be applied to devices with… Figure 4 The hardware structure shown and Figure 5 In the electronic device with the software architecture shown, taking a mobile phone as an example, as follows... Figure 6 As shown, the shooting method provided in this application embodiment may include S601 to S604, which are described in detail below:

[0124] S601, the electronic device displays the shooting preview interface.

[0125] Electronic devices can have a camera application installed. Upon receiving a camera launch command, the electronic device can display a shooting preview interface. This camera launch command can be used to instruct the user to launch the camera application.

[0126] For example, when an electronic device receives a camera activation command, it may be in situations including but not limited to the following:

[0127] Scenario 1.1: The electronic device detects the first action targeting the camera app icon.

[0128] For example, the first operation can be a click operation or a long press operation, etc.

[0129] In practical applications, the camera application icon can be included in multiple display interfaces of an electronic device. These multiple display interfaces may include, for example, the desktop, the negative one screen, the control center, the lock screen, or interfaces of other applications. Other applications can refer to any application on the electronic device other than the camera application, such as social media applications or image processing applications.

[0130] Based on this, for example, an electronic device can determine that it has received a camera launch command when it detects that the camera app icon on the desktop has been clicked. For another example, an electronic device can determine that it has received a camera launch command when it detects that the camera app icon on the -1 screen has been clicked. For yet another example, an electronic device can determine that it has received a camera launch command when it detects that the camera app icon in the Control Center has been clicked. For yet another example, an electronic device can determine that it has received a camera launch command when it detects that the camera app icon on the lock screen has been clicked or long-pressed. For yet another example, an electronic device can determine that it has received a camera launch command when it detects that the camera app icon in other application interfaces has been clicked.

[0131] In scenario 1.2, the electronic device detects a camera quick start operation.

[0132] For example, quick camera launch operations may include, but are not limited to: swiping up from the bottom left or bottom right corner of the lock screen; or pressing the same volume button twice consecutively while the screen is locked.

[0133] Based on this, for example, an electronic device can determine that it has received a camera launch command when it detects that a user swipes up from the bottom left or bottom right corner of the lock screen. As another example, an electronic device can determine that it has received a camera launch command when it detects that the same volume button is pressed twice consecutively while the screen is locked.

[0134] In scenario 1.3, the electronic device receives a voice command to launch the camera application.

[0135] For example, the voice command to launch the camera app could be "Xiaoyi, open the camera". Based on this, the electronic device can determine that it has received a camera launch command when it receives the voice command "Xiaoyi, open the camera".

[0136] For example, please refer to Figure 7 This is a schematic diagram of a shooting preview interface provided in an embodiment of this application. Figure 7 As shown in (a) above, the shooting preview interface may include a shooting preview frame 701 and shooting controls 702, etc. Figure 7 As shown in (b) above, the shooting preview box 701 can be used to display a preview image 7011. The preview image 7011 may include the scene to be shot captured by the camera of the electronic device, such as a fireworks scene. The shooting control 702 can be used to control the electronic device to start or stop shooting, etc.

[0137] S602, when an electronic device receives a shooting instruction and the scene to be shot in the shooting preview frame is the target scene, it captures at least one set of images using multiple cameras.

[0138] For example, when an electronic device receives a shooting instruction, it may be in situations including but not limited to the following:

[0139] Scenario 2.1: The electronic device detects a second operation on the shooting controls in the shooting preview interface.

[0140] For example, the second operation can be a click operation or a repeated press operation. Based on this, the electronic device can determine that it has received a shooting command when it detects that the shooting control in the shooting preview frame has been clicked or repeatedly pressed.

[0141] In scenario 2.2, the electronic device detects a quick shooting operation while displaying the shooting preview interface.

[0142] For example, quick shooting operations may include, but are not limited to: pressing a preset volume button, performing a preset air gesture, or clicking any location in the shooting preview frame. The preset volume button can be a volume down button or a volume up button. Preset air gesture operations may include an air grasping gesture or an air V-sign gesture.

[0143] In practical applications, the quick shooting operation can be configured by the user according to their own needs.

[0144] Based on this, for example, if a user sets pressing the volume down button as the quick capture button, the electronic device can determine that a capture command has been received when the volume down button is pressed, while the capture preview interface is displayed. Similarly, if a user sets pressing the volume up button as the quick capture button, the electronic device can determine that a capture command has been received when the volume up button is pressed, while the capture preview interface is displayed. Furthermore, if a user sets an air gesture as the quick capture operation, the electronic device can determine that a capture command has been received when the air gesture is detected, while the capture preview interface is displayed. Similarly, if a user sets an air V-sign as the quick capture operation, the electronic device can determine that a capture command has been received when the air V-sign is detected, while the capture preview interface is displayed. Finally, if a user sets clicking any location in the capture preview frame as the quick capture operation, the electronic device can determine that a capture command has been received when any location in the capture preview frame is clicked, while the capture preview interface is displayed.

[0145] In scenario 2.3, the electronic device receives a voice command to take a picture while displaying the shooting preview interface.

[0146] For example, the voice command to take a picture could be "take a picture" or "take a video".

[0147] Based on this, electronic devices can confirm receipt of a shooting command when they receive a voice shooting command such as "take a photo" or "shoot a video" while displaying a shooting preview interface.

[0148] Scenario 2.4: When an electronic device is displaying a shooting preview interface, it receives a shooting instruction from another electronic device.

[0149] Other electronic devices can be any electronic device that has established a communication connection with the electronic device. For example, other electronic devices may include smartwatches or smart bracelets. The communication connection can be a wired connection or a wireless connection. A wired connection may, for example, include a USB-based connection. A wireless connection may, for example, include a Bluetooth connection or a Wi-Fi connection.

[0150] For example, other electronic devices may be equipped with a shutter button. Based on this, when the shutter button on another electronic device is pressed, the other electronic device can send a shutter command to the electronic device.

[0151] In some embodiments, the electronic device can perform target recognition on the scene to be shot in the shooting preview frame in real time after displaying the shooting preview interface, so as to determine whether the scene to be shot is the target scene.

[0152] In other embodiments, the electronic device may perform target recognition on the scene to be photographed in the shooting preview frame after receiving the shooting instruction, so as to determine whether the scene to be photographed is the target scene.

[0153] For example, electronic devices can employ target recognition algorithms to identify whether the scene to be photographed is a target scene. It should be noted that the specific recognition process of the target recognition algorithm can be found in descriptions in related technologies, and will not be detailed here.

[0154] For example, the target scene can refer to a high-brightness scene that changes dynamically at night. For instance, the target scene may include a fireworks scene or a scene of light sources moving rapidly at night (such as car headlights or scrolling signs).

[0155] Taking a fireworks scene as an example, the electronic device can take at least one set of images and execute subsequent steps S603 to S604 after receiving a shooting instruction and recognizing that the scene to be shot in the shooting preview frame is a fireworks scene.

[0156] Each set of images can include multiple images. These multiple images can be multiple images of the scene being filmed, captured simultaneously by an electronic device using multiple different cameras, with the same exposure time but different exposure levels. In other words, the multiple images in each set are captured at the same time, have the same exposure time, and have different exposure levels.

[0157] For example, when an electronic device captures each set of images, it can use the camera with better light sensitivity among multiple cameras to capture images with lower exposure, and use the camera with poorer light sensitivity among multiple cameras to capture images with higher exposure. That is, in each set of images, images with lower exposure can be captured by the camera with better light sensitivity, and images with higher exposure can be captured by the camera with poorer light sensitivity.

[0158] It's important to note that "good light sensitivity" and "poor light sensitivity" are relative terms. Specifically, good light sensitivity means the image sensor in the camera is generally more sensitive to light. Poor light sensitivity means the image sensor in the camera is generally less sensitive to light. For example, in a multi-camera setup including a main camera and a telephoto camera, since the image sensor in the main camera is generally more sensitive to light than the image sensor in the telephoto camera, the main camera is considered to have good light sensitivity, while the telephoto camera is considered to have poor light sensitivity. Similarly, low exposure and high exposure are also relative terms. For instance, assuming each set of images includes a first exposure image and a second exposure image, and the first exposure image has a lower exposure than the second exposure image, then the first exposure image in each set is considered to have lower exposure, and the second exposure image is considered to have higher exposure.

[0159] It should also be noted that the shooting times for different sets of images are different. For example, a preset time interval can be maintained between the shooting times of each two adjacent sets of images; that is, the electronic device can take a set of images every preset time interval. The preset time interval can be set according to actual needs. For example, the preset time interval can be 0.1 seconds.

[0160] In practical applications, the field of view of multiple different cameras used to capture each set of images can be all the same, partially the same, or all different. For example, in some embodiments, the multiple different cameras may include a main camera and a telephoto camera. In other embodiments, the multiple different cameras may include a main camera and a wide-angle camera. In still other embodiments, the multiple different cameras may include two main cameras with different performance characteristics. In yet another embodiment, the multiple different cameras may include a telephoto camera, a main camera, and a wide-angle camera.

[0161] For example, please refer to Figure 8 This is a schematic diagram of multiple images captured by an electronic device based on the shooting method provided in the embodiments of this application. Figure 8As shown, taking multiple different cameras, including a main camera and a telephoto camera, an electronic device captures a first exposure image using the main camera and a second exposure image using the telephoto camera, with the exposure of the first exposure image being less than that of the second exposure image. When the electronic device receives a shooting command and recognizes the scene to be captured in the shooting preview frame as the target scene, it can simultaneously capture a set of images (i.e., ...) using the main camera and the telephoto camera at preset time intervals. Figure 8 (The first exposure image and the second exposure image in each dashed box in the image are a set of images, thus obtaining at least one set of images.)

[0162] Understandably, since the exposure of an image is determined by the aperture, exposure time, and ISO of the camera that captured the image, electronic devices can control the exposure of each image in each group to be different, while ensuring that the exposure time of multiple images in each group is the same:

[0163] Method 1: For electronic devices with fixed aperture values ​​and adjustable ISO values, the electronic device can control the ISO value of a camera with better light sensitivity to be lower than the ISO value of a camera with poorer light sensitivity.

[0164] For example, taking multiple cameras including a main camera and a telephoto camera, where the main camera is used to capture an image with a first exposure value and the telephoto camera is used to capture an image with a second exposure value, and the exposure value of the first exposure image is less than the exposure value of the second exposure image, the electronic device can control the exposure value of the first exposure image to be less than the exposure value of the second exposure image by controlling the ISO value of the main camera to be less than the ISO value of the telephoto camera.

[0165] Method 2: For electronic devices with adjustable aperture values ​​and fixed ISO values, the electronic device can control the aperture value of the camera with better light sensitivity to be greater than that of the camera with poorer light sensitivity.

[0166] For example, taking multiple cameras including a main camera and a telephoto camera, where the main camera is used to capture an image with a first exposure value and the telephoto camera is used to capture an image with a second exposure value, and the exposure value of the first exposure image is less than that of the second exposure image, the electronic device can control the exposure value of the first exposure image to be less than that of the second exposure image by controlling the aperture value of the main camera to be greater than that of the telephoto camera.

[0167] Method 3: For electronic devices where both the aperture and ISO values ​​of the camera are adjustable, the electronic device can control the ISO value of the camera with better light sensitivity to be lower than that of the camera with poorer light sensitivity, and / or control the aperture value of the camera with better light sensitivity to be greater than that of the camera with poorer light sensitivity.

[0168] For example, taking multiple cameras including a main camera and a telephoto camera, where the main camera is used to capture an image with a first exposure value and the telephoto camera is used to capture an image with a second exposure value, and the exposure value of the first exposure image is less than the exposure value of the second exposure image, the electronic device can control the exposure value of the first exposure image to be less than the exposure value of the second exposure image by controlling the ISO value of the main camera to be less than the ISO value of the telephoto camera, and / or controlling the aperture value of the main camera to be greater than the aperture value of the telephoto camera.

[0169] It is understandable that telephoto cameras are more prone to insufficient light intake compared to main cameras. Therefore, in this embodiment, a first exposure image with a smaller exposure is captured by a main camera with better light sensitivity, and a second exposure image with a larger exposure is captured by a telephoto camera with poorer light sensitivity. This can simultaneously improve the signal-to-noise ratio of the first exposure image and the signal-to-noise ratio of the second exposure image, thereby enabling the electronic device to fuse the first exposure image and the second exposure image into a higher quality fused image.

[0170] In practical applications, when electronic devices capture images from multiple different cameras simultaneously, in order to ensure that the multiple images in each group are captured at the same time, the electronic devices can use software synchronization control or hardware synchronization control to control the multiple cameras to capture images at the same time.

[0171] In one specific implementation, the electronic device uses software synchronous control to control multiple cameras to capture images simultaneously. This can include the electronic device sending shooting commands to multiple cameras simultaneously via a processor. This ensures that multiple cameras can receive the shooting commands at the same time and capture images concurrently.

[0172] In another specific implementation, a hardware synchronization circuit can be set up between multiple cameras of the electronic device. This hardware synchronization circuit is used to control multiple cameras to perform shooting operations simultaneously when a shooting command is received. Based on this, the electronic device uses hardware synchronization control to control multiple cameras to shoot simultaneously, which can include: the electronic device sends a shooting command to the hardware synchronization circuit through the processor to trigger the hardware synchronization circuit to control multiple cameras to perform shooting operations simultaneously.

[0173] S603, the electronic device performs a first processing on each group of images to make the field of view, size and resolution of all images in each group the same; each group of images after the first processing is a group of images to be fused.

[0174] It is understandable that, since the multiple images in each group of images may have been captured by the electronic device through multiple cameras with the same field of view, or multiple cameras with partially the same field of view, or multiple cameras with different field of view, in order to ensure that the field of view, size, and resolution of each image used for subsequent image fusion are the same, thereby improving the quality of the fused image, in some embodiments, when the field of view of the multiple images in each group of images is not completely the same, the electronic device may perform a first processing on each group of images to make the field of view, size, and resolution of the multiple images in each group of images the same.

[0175] In a specific implementation, the first process may include cropping, scaling, and upsampling. Based on this, such as... Figure 9 As shown, the electronic device performs a first process on each group of images, which may include steps S6031 to S6033, as detailed below:

[0176] S6031, the electronic device crops the large field-of-view image in each group of images to obtain the cropped image corresponding to the large field-of-view image; the field of view corresponding to the cropped image is the same as the field of view corresponding to the small field-of-view image.

[0177] It should be noted that the terms "wide field of view image" and "narrow field of view image" are relative. Specifically, a wide field of view image refers to an image captured by the camera with the largest field of view among multiple cameras, while a narrow field of view image refers to an image captured by the camera with the smallest field of view among multiple cameras. For example, suppose that the first exposure image in each set of images is captured by the main camera and the second exposure image is captured by the telephoto camera. Since the field of view of the main camera is larger than that of the telephoto camera, the first exposure image in each set of images is a wide field of view image, and the second exposure image is a narrow field of view image.

[0178] For example, please refer to Figure 10 This is a schematic diagram illustrating the processing steps involved in the first processing of each group of images by an electronic device according to an embodiment of this application. Figure 10As shown, assuming each set of images includes a first exposure image and a second exposure image, the first exposure image is captured by the main camera, and the second exposure image is captured by the telephoto camera, that is, the first exposure image is a large field of view image, and the second exposure image is a small field of view image; and the shadow portion in the first exposure image is the overlapping portion of the field of view corresponding to the first exposure image and the field of view corresponding to the second exposure image, that is, the field of view corresponding to the shadow portion in the first exposure image is the same as the field of view corresponding to the second exposure image, then the electronic device can crop the first exposure image in each set of images according to the field of view corresponding to the second exposure image in each set of images, so as to crop out the shadow portion from the first exposure image. This shadow portion is the cropped image corresponding to the first exposure image, and the field of view corresponding to the cropped image is the same as the field of view corresponding to the second exposure image.

[0179] S6032, the electronic device scales the cropped image corresponding to the large field of view image in each group of images to obtain the scaled image corresponding to the large field of view image; the size of the scaled image is the same as the size of the small field of view image, and the field of view corresponding to the scaled image is the same as the field of view corresponding to the small field of view image.

[0180] The image size can refer to the actual size of the image in physical space. For example, the image size can be represented by its width and height. For instance, an image size of 25.6 × 34.14 cm can be used to represent an image with a width of 25.6 cm and a height of 34.14 cm.

[0181] It is understandable that, since the original size of each image in each group is the same, and the size of the cropped image obtained by the electronic device after cropping the large field-of-view image is smaller than the size of the small field-of-view image, the electronic device can scale the cropped image corresponding to the large field-of-view image to make the size of the scaled cropped image the same as the size of the small field-of-view image in the same group in order to facilitate subsequent image fusion. For ease of explanation, the scaled cropped image in this application embodiment is described as the scaled image corresponding to the large field-of-view image.

[0182] Specifically, scaling the cropped image corresponding to the large field-of-view image by the electronic device may include: scaling the width of the cropped image to be equal to the width of the small field-of-view image, and scaling the height of the cropped image to be equal to the height of the small field-of-view image.

[0183] For example, please continue reading Figure 10Assuming each set of images includes a first exposure image and a second exposure image, where the first exposure image is captured by the main camera and the second exposure image is captured by the telephoto camera, after the electronic device crops the image corresponding to the first exposure image in each set, it can scale the cropped image corresponding to the first exposure image to obtain a scaled image corresponding to the first exposure image. The size of this scaled image is the same as the size of the second exposure image, and the field of view of this scaled image is the same as the field of view of the second exposure image.

[0184] S6033, the electronic device upsamples the scaled image corresponding to the large field of view image in each group of images to obtain the upsampled image corresponding to the large field of view image; the resolution of the upsampled image is the same as the resolution of the small field of view image, and the size of the upsampled image is the same as the size of the small field of view image, and the field of view corresponding to the upsampled image is the same as the field of view corresponding to the small field of view image.

[0185] The resolution of an image can be used to represent the distribution of pixels in the width and height of the image. For example, the resolution of an image can be 1920×1080 pixels, which can be used to represent that the image has 1920 pixels in the width direction (i.e., the horizontal direction) and 1080 pixels in the height direction (i.e., the vertical direction).

[0186] It is understandable that although the size of the scaled image corresponding to the large field-of-view image in each group of images is the same as the size of the small field-of-view image in the same group, the resolution of the cropped image is lower than that of the small field-of-view image because the cropped image is cropped from the large field-of-view image. Furthermore, the scaled image only stretches the width and height of the cropped image and does not increase its resolution, thus further reducing the resolution of the scaled image to be lower than that of the small field-of-view image. Therefore, to facilitate subsequent image fusion, the electronic device can upsample the scaled image corresponding to the large field-of-view image in each group of images so that the resolution of the upsampled scaled image is the same as that of the small field-of-view image. For ease of explanation, this embodiment describes the upsampled scaled image as the upsampled image corresponding to the large field-of-view image.

[0187] For example, electronic devices can use interpolation methods to upsample scaled images. These interpolation methods can include nearest-neighbor interpolation, bilinear interpolation, and Lagrange interpolation, among others. It should be noted that the specific operational procedures for interpolation methods can be found in descriptions in related technologies, and will not be detailed here.

[0188] For example, electronic devices can upsample scaled images based on super-resolution technology. It should be noted that the specific operational procedures of super-resolution technology can be found in descriptions in related technologies, and will not be detailed here.

[0189] Understandably, for ease of explanation, such as Figure 8 As shown in the embodiments of this application, each group of images that has undergone the first processing can be described as a group of images to be fused.

[0190] It is also understandable that, in specific applications, due to differences in the manufacturing processes and optical characteristics of different cameras, there are different distortions (i.e., geometric distortion or other distortions) between images captured by different cameras. Therefore, in order to align the images captured by different cameras onto the same plane and reduce or eliminate the distortion differences between the images captured by different cameras, in another specific implementation, the first process may also include: distortion correction.

[0191] Based on this, such as Figure 11 As shown, prior to S6031, S603 may further include S6034, as detailed below:

[0192] S6034, the electronic device performs distortion correction on each image based on the calibration intrinsic parameters of the camera corresponding to each image.

[0193] The camera corresponding to the image can refer to the camera used to capture that image.

[0194] The calibration parameters of a camera can refer to the pre-calibrated internal parameters of the camera.

[0195] For example, the internal parameters of a camera may include its distortion coefficients and focal length. The distortion coefficients may include radial distortion coefficients and tangential distortion coefficients. The radial distortion coefficient describes the degree to which individual pixels in an image are bent outwards or inwards as they move further from the center pixel. The tangential distortion coefficient describes the degree of skewness of individual pixels in the image caused by the camera not being aligned with the image plane. The focal length of the camera may include its horizontal focal length and its vertical focal length.

[0196] For example, S6034 may specifically include steps 1.1 to 1.2, as detailed below:

[0197] Step 1.1: The electronic device determines the intrinsic parameter matrix of the camera corresponding to each image based on the coordinates of the center pixel of each image and the focal length of the camera corresponding to each image.

[0198] The intrinsic parameter matrix of the camera can be used to describe the camera's inherent characteristics.

[0199] For example, the intrinsic parameter matrix of the camera corresponding to each image can be represented as:

[0200]

[0201] Where K is the intrinsic parameter matrix of the camera, f x f represents the focal length of the camera in the horizontal direction. y c represents the focal length of the camera in the vertical direction. x c represents the x-coordinate of the center pixel of the image. y The vertical coordinate of the center pixel of the image.

[0202] Step 1.2: The electronic device performs distortion correction on each image based on the intrinsic parameter matrix and distortion coefficients of the camera corresponding to each image, thereby obtaining the distortion-corrected image corresponding to each image.

[0203] For example, an electronic device can calculate the degree of distortion (i.e., offset) of each pixel in each image based on the distortion coefficient of the camera corresponding to each image; determine the new coordinates of each pixel based on the original coordinates and the degree of distortion of each pixel in each image; and determine the actual position of each pixel in the world coordinate system based on the new coordinates of each pixel in each image and the intrinsic parameter matrix of the camera corresponding to each image, thereby obtaining the distortion-corrected image corresponding to each image.

[0204] This application embodiment corrects the distortion of each image in each group of images, thereby correcting images captured by different cameras to the same reference plane. This reduces or eliminates the distortion differences between images in each group of images, improving the registration accuracy during subsequent image registration.

[0205] It is also understandable that in specific applications, the installation positions and / or installation angles of multiple cameras on electronic devices cannot be completely coincident. This would cause the images captured by different cameras to be unable to be aligned to the same reference coordinate system. Therefore, in order to reduce the difference in perspective between different images caused by the different installation positions and / or installation angles of different cameras, in another specific implementation, the first process may also include: affine transformation.

[0206] Based on this, please continue reading Figure 11 Prior to S6031, S603 may also include S6035, as detailed below:

[0207] S6035, the electronic device performs affine transformation on each image based on the calibration extrinsic parameters of the camera corresponding to each image.

[0208] The calibration extrinsic parameters of a camera refer to the pre-calibrated external parameters of the camera. These external parameters can be used to describe the camera's pose, for example, its pose relative to the world coordinate system.

[0209] For example, the external parameters of the camera may include: the translation of the camera in the three degrees of freedom of the spatial coordinate system (i.e., the x-axis, y-axis and z-axis), and the rotation angle of the camera around the three degrees of freedom, etc.

[0210] For example, S6035 may specifically include steps 2.1 to 2.4, as detailed below:

[0211] Step 2.1: The electronic device selects several feature points from each image in each group of images.

[0212] For example, the aforementioned feature points can be corner points, edge points, or other prominent feature points of the image.

[0213] For example, electronic devices can use feature point detection algorithms to select several feature points from each image. Feature point detection algorithms may include, for example, scale-invariant feature transform (SIFT) algorithms or speeded-up robust features (SURF) algorithms.

[0214] Step 2.2: The electronic device uses a feature point matching algorithm to match the feature points of different images in each group of images, resulting in multiple feature point pairs.

[0215] For example, feature matching algorithms may include, but are not limited to, the K nearest neighbors (KNN) algorithm.

[0216] Step 2.3: The electronic device determines the affine transformation matrix based on the coordinates of multiple feature point pairs.

[0217] For example, an electronic device can determine the affine transformation matrix based on the coordinates of multiple feature point pairs using an affine transformation function. An affine transformation function could, for example, include the `cv2.getAffineTransform()` function from the OpenCV library. Details regarding the `cv2.getAffineTransform()` function can be found in related technical documents and will not be elaborated upon here.

[0218] For example, the affine transformation matrix can be represented as:

[0219]

[0220] Where M is the affine transformation matrix, a 11 a represents the scaling of pixels in an image along the x-axis. 22 a represents the scaling of pixels in an image along the y-axis. 12 a represents the amount of pixel clipping along the x-axis in an image. 21 t represents the amount of pixel clipping along the y-axis in an image. x t represents the translation of a pixel on the x-axis in an image. y This represents the amount of translation of a pixel in the image along the y-axis.

[0221] Step 2.4: The electronic device performs an affine transformation on each image in each group of images based on the affine transformation matrix to obtain the affine transformed image corresponding to each image.

[0222] For example, the electronic device can process each image in each group of images based on the affine transformation matrix using the following formula (2):

[0223]

[0224] Among them, (x i ,y i Let (x) be the coordinates of the i-th pixel in each image. i ',y i ') represents the coordinates of the i-th pixel in each image after affine transformation.

[0225] This application embodiment aligns all images in each group of images to a unified reference coordinate system by performing an affine transformation on each image in each group of images. This reduces or eliminates the physical viewpoint difference between images in each group of images, thereby improving the registration accuracy during subsequent image registration.

[0226] It is also understandable that, in practical applications, when electronic devices use software synchronization control to control multiple cameras to shoot simultaneously, there may still be millimeter-level errors between the shooting times of different cameras. Therefore, in order to ensure that the content of images captured by different cameras in the same set of images is aligned as much as possible, the first process may also include: inter-group registration. Inter-group registration can refer to registering multiple frames within the same set of images.

[0227] Based on this, please continue reading Figure 11 Prior to S6031, S603 may also include S6036, as detailed below:

[0228] S6036, the electronic device performs inter-group registration on multiple images in each group of images.

[0229] For example, an electronic device can use the image with lower exposure in each group of images as a reference frame, and align the image with higher exposure in the same group to the image with lower exposure, thereby achieving registration between multiple images in the same group. Aligning the image with higher exposure to the image with lower exposure can include aligning each pixel in the image with a corresponding pixel (i.e., a pixel with identical content) in the image with lower exposure.

[0230] It should be noted that S6034, S6035 and S6036 can be placed before S6031. That is, the electronic device can execute S6034 and / or S6035 and / or S6036 first, and then execute S6031 to S6033.

[0231] This application embodiment, by performing inter-group registration on each group of images, can align the content of images taken by different cameras at different exposure levels as much as possible, thereby improving the subsequent image fusion effect.

[0232] In yet another specific implementation, the first process may also include: brightness alignment.

[0233] Based on this, please continue reading Figure 11 Following S6033, S603 may also include S6037, as detailed below:

[0234] S6037, the electronic device calculates the difference in exposure value between every two images in each group of images, and adjusts the brightness of each pair of images based on the difference in exposure value between each pair of images, so that the brightness of all images in each group of images is the same.

[0235] It should be noted that S6037 can be located after S6031 to S6033, that is, the electronic device can execute S6031 to S6033 first, and then execute S6037.

[0236] This application embodiment aligns the brightness of each image in each group of images, ensuring that the brightness of all images in each group is consistent, which is beneficial for improving the subsequent image fusion effect.

[0237] In other embodiments of this application, the electronic device may further perform a second processing on all groups of images. Exemplarily, the second processing may include inter-frame registration. Inter-frame registration may refer to registering all images in all groups of images.

[0238] Based on this, after S603 and before S604, the shooting method can also include step 3.1, detailed below:

[0239] Step 3.1: The electronic device performs inter-frame registration on all images in all groups of images.

[0240] For example, an electronic device can use one image from a set of images as a reference image, calculate the registration matrix between the other images and the reference image, and register each other image based on the registration matrix between each other image and the reference image, so as to achieve registration between all images in all sets of images.

[0241] This application embodiment can further align all images by registering all images in all groups of images, thereby further improving the subsequent image fusion effect.

[0242] Based on this, in some embodiments, if the shooting method does not include step 3.1 above, the electronic device can directly determine each group of images after the first processing as a group of images to be fused.

[0243] In other embodiments, when the shooting method includes step 3.1 above, the electronic device can determine each group of images that have undergone the first and second processing as a group of images to be fused.

[0244] S604, the electronic device inputs multiple groups of images to be fused that are adjacent in time of capture into a trained image denoising and fusion model for processing, and obtains a fused image.

[0245] The image denoising and fusion model can include multiple denoising units and one image fusion unit. The total number of denoising units can be equal to the total number of cameras used when an electronic device captures multiple sets of images, and each denoising unit can correspond one-to-one with a different camera. For example, the input to each denoising unit can be multiple images taken by its corresponding camera at the same exposure level, and the output of each denoising unit can be connected to the image fusion unit. Based on this, each denoising unit can be used to fuse and denoise multiple images taken by its corresponding camera at the same exposure level, and output a denoised image at the corresponding exposure level to the fusion unit. The fusion unit can be used to fuse multiple denoised images from multiple denoising units at different exposure levels to obtain a fused image.

[0246] Based on this, S604 may specifically include steps 4.1 to 4.2, as detailed below:

[0247] Step 4.1: The electronic device uses each denoising unit in the image denoising fusion model to fuse and denoise multiple images with the same exposure taken by the camera corresponding to each denoising unit, and outputs a denoised image with the corresponding exposure to the image fusion unit.

[0248] Step 4.2: The electronic device fuses multiple denoised images from multiple denoising units under different exposure levels through the image fusion unit to obtain a fused image.

[0249] For ease of understanding, the following example illustrates the specific structure of the image denoising and fusion model, where each image set includes a first exposure image and a second exposure image, with the first exposure image captured by the main camera and the second exposure image captured by the telephoto camera. Please refer to [link / reference]. Figure 12 This is a schematic diagram of the structure of an image denoising and fusion model provided in an embodiment of this application.

[0250] like Figure 12 As shown, the image denoising and fusion model may include a first denoising unit 1201, a second denoising unit 1202, and an image fusion unit 1203. The first denoising unit 1201 may correspond to the main camera, and the second denoising unit 1202 may correspond to the telephoto camera. Specifically, the first denoising unit 1201 can fuse and denoise multiple first exposure images captured by the main camera to obtain a denoised image corresponding to the first exposure image, and output the denoised image corresponding to the first exposure image to the fusion unit 1203. Specifically, the second denoising unit 1202 can fuse and denoise multiple second exposure images captured by the telephoto camera to obtain a denoised image corresponding to the second exposure image, and output the denoised image corresponding to the second exposure image to the fusion unit 1203. Specifically, the fusion unit 1203 can fuse the denoised image corresponding to the first exposure image and the denoised image corresponding to the second exposure image to obtain a fused image.

[0251] Based on this, the electronic device can use the first denoising unit 1201 to fuse and denoise multiple first exposure images captured by the main camera, and output a denoised image corresponding to the first exposure image to the image fusion unit 1203; and can use the second denoising unit 1202 to fuse and denoise multiple second exposure images captured by the telephoto camera, and output a denoised image corresponding to the second exposure image to the image fusion unit 1203; and can use the image fusion unit 1203 to fuse the denoised image corresponding to the first exposure image and the denoised image corresponding to the second exposure image to obtain a fused image.

[0252] For example, the structure of each denoising unit in the image denoising fusion model can adopt a deep learning-based convolutional neural network structure. For example, the convolutional neural network structure can include one or more attention-based convolutional blocks (transformer blocks) and one or more ordinary convolutional blocks (conv).

[0253] The image fusion unit in the image denoising and fusion model can employ the HDR multi-frame fusion module from related technologies. For details regarding the HDR multi-frame fusion module, please refer to the descriptions in related technologies; they will not be elaborated upon here.

[0254] It is understandable that, since different denoising units in an image denoising fusion model are used to denoise images captured by different cameras, and different cameras have different noise coefficients, the images captured by different cameras will carry different levels of noise. Therefore, different denoising units need to have different denoising capabilities. Based on this, in practical applications, different denoising units can be trained using different training datasets when training the image denoising fusion model. For example, using... Figure 12 Taking the image denoising fusion model shown as an example, the first denoising unit 1201 can be trained using the first training dataset, and the second denoising unit 1202 can be trained using the second training dataset.

[0255] In other words, different denoising units in the image denoising fusion model are trained using different training datasets. For example, the training dataset corresponding to each denoising unit can include multiple sample data points. Each sample data point can include multiple noisy images and one noise-free real image. These multiple noisy images can be obtained by adding noise to the real image. Based on this, in specific applications, when training the image denoising fusion model, the electronic device can use multiple noisy images from each sample data point corresponding to each noise unit as the input of that noise unit, and use one real image from each sample data point as the output of that noise unit, training each noise unit separately, so that each noise unit learns different denoising capabilities.

[0256] For example, please refer to Figure 13 This is a schematic diagram illustrating the generation process of the training dataset corresponding to the denoising unit in an image denoising and fusion model provided in this application embodiment. Figure 13 As shown, the electronic device can generate the training dataset corresponding to each denoising unit through S1301 to S1308, as detailed below:

[0257] S1301, the electronic device acquires multiple high-definition night scene images.

[0258] The high-definition night scene image can be any night scene image captured by professional shooting equipment (such as an SLR camera or a point-and-shoot camera). For example, the high-definition night scene image can be an image in RGB format.

[0259] S1302, for each high-definition night scene image, the electronic device randomly crops out a region from the high-definition night scene image as the image to be degraded.

[0260] S1303, the electronic device performs random preprocessing on the image to be degraded.

[0261] For example, random preprocessing may include, but is not limited to: random white balance processing, random rotation, random addition of ghosting, random addition of color blocks, and random addition of jitter blur.

[0262] The purpose of random white balance processing on the image to be degraded by the electronic device is to randomly modify the color temperature of the image. For example, the electronic device can use any white balance coefficient to perform white balance processing on the image to be degraded.

[0263] It should be noted that operations such as rotating, sharpening, adding ghosting, adding color blocks, and adding jitter blur are all common image processing methods in the field of image processing. For the specific processing procedures of these image processing methods, please refer to the descriptions in relevant technologies, which will not be detailed here.

[0264] S1304, the electronic device determines the degraded image that has undergone random preprocessing as a real image.

[0265] S1305, an electronic device reproduces multiple copies of a real image.

[0266] The total number of duplicate images can be equal to the total number of cameras used when an electronic device captures multiple sets of images. Each duplicate image can correspond one-to-one with a different camera.

[0267] S1306, the electronic device performs a first noise-adding process on the copy image corresponding to the wide field-of-view camera to obtain a noisy image corresponding to the wide field-of-view camera.

[0268] In this context, a wide field-of-view camera can refer to a camera with a relatively large field of view among multiple cameras. It's understandable that an image captured by a wide field-of-view camera can be called a wide field-of-view image. For example, the first exposure image captured by the main camera can be considered a wide field-of-view image.

[0269] For example, the noise addition process corresponding to the first noise addition process may include: image blurring, sensitivity compensation, adding first noise corresponding to a wide field of view camera, and format conversion, etc.

[0270] Based on this, S1306 may specifically include S13061 to S13064, as detailed below:

[0271] S13061, Electronic device performs image blurring processing on the copy image corresponding to the wide field of view camera.

[0272] It is understandable that since the electronic device will crop the wide field of view image in the aforementioned S6031, there will be image quality blur in the wide field of view image due to image cropping. Based on this, in order to more realistically simulate the image input to the noise unit corresponding to the wide field of view camera, the electronic device can perform image quality blur processing on the copy image corresponding to the wide field of view camera to add image quality blur introduced by image cropping.

[0273] For example, an electronic device can perform a random downsampling operation on a copy image corresponding to a wide field-of-view camera, and then perform a random upsampling operation on the copy image after the random downsampling operation, in order to add image blur caused by image cropping to the copy image corresponding to the wide field-of-view camera.

[0274] S13062, Electronic device performs photosensitivity compensation on a copy image that has undergone image quality blurring.

[0275] Understandably, since different cameras have different light sensitivity, in order to simulate the difference in light sensitivity between different cameras, electronic devices can perform light sensitivity compensation on the blurred copy image based on the difference in light sensitivity between different cameras.

[0276] For example, an electronic device can multiply the pixel value of each pixel in a copy image that has undergone image quality blurring by a sensitivity difference to achieve sensitivity compensation for the copy image corresponding to a wide field of view camera.

[0277] S13063, the electronic device adds first noise to a photosensitive copy image based on the noise figure of a wide field-of-view camera.

[0278] The first noise may include a first Gaussian white noise and a first Poisson particle noise corresponding to the noise figure of the wide field-of-view camera.

[0279] S13064, the electronic device converts the copy image with added first noise into a raw format image to obtain the noisy image corresponding to the wide field of view camera.

[0280] The raw format can also be understood as the original format.

[0281] It is understandable that since high-definition night scene images are in RGB format, the real images obtained from high-definition night scene images are also in RGB format, and the copy images of the real images are also in RGB format. However, in specific applications, the images input into the image noise fusion model are the original format (i.e., raw format) images captured by the camera. Therefore, electronic devices need to convert the copy images corresponding to the large field-of-view camera with added first noise into raw format images. These raw format images are the noise images corresponding to the large field-of-view camera.

[0282] S1307, the electronic device performs a second noise-adding process on the copy image corresponding to the small field-of-view camera to obtain a noisy image corresponding to the small field-of-view camera.

[0283] In this context, a small field-of-view camera can refer to a camera with a relatively small field of view among multiple cameras. It's understood that an image captured by a small field-of-view camera can be called a small field-of-view image. For example, a second-exposure image captured by a telephoto camera can be described as a small field-of-view image.

[0284] The noise addition process for the second noise addition process is different from the noise addition process for the first noise addition process.

[0285] For example, the noise addition process corresponding to the second noise addition process may include: exposure brightening, adding second noise corresponding to the small field of view camera, and format conversion, etc.

[0286] Based on this, S1307 may specifically include S13071 to S13073, as detailed below:

[0287] S13071, The electronic device brightens the copy image corresponding to the small field-of-view camera.

[0288] For example, an electronic device can randomly brighten a copy of the image from a small field-of-view camera to simulate overexposure in an image captured by the small field-of-view camera.

[0289] S13072, an electronic device adds a second noise to a brightened copy image based on the noise figure of a small field-of-view camera.

[0290] The second noise may include second Gaussian white noise and second Poisson particle noise corresponding to the noise figure of the small field-of-view camera.

[0291] S13073, the electronic device converts the copy image with added second noise into a raw format image to obtain the noisy image corresponding to the small field of view camera.

[0292] It is understandable that since high-definition night scene images are in RGB format, the real images obtained from high-definition night scene images are also in RGB format, and the copy images of the real images are also in RGB format. However, in specific applications, the images input into the image noise fusion model are the original format (i.e., raw format) images captured by the camera. Therefore, electronic devices need to convert the copy images corresponding to the small field-of-view camera with added second noise into raw format images. These raw format images are the noise images corresponding to the small field-of-view camera.

[0293] S1038 takes the noise image and the real image corresponding to the large field of view camera obtained from each high-definition night scene image as a sample data of the denoising unit corresponding to the large field of view camera; and takes the noise image and the real image corresponding to the small field of view camera obtained from each high-definition night scene image as a sample data of the denoising unit corresponding to the small field of view camera.

[0294] Since electronic devices can obtain one sample data point corresponding to each denoising unit from each high-definition night scene image, the training dataset corresponding to each denoising unit can be obtained from multiple high-definition night scene images.

[0295] In this embodiment, since different denoising units in the image denoising fusion model are trained using a training dataset that matches their required denoising capabilities, different denoising units can have different denoising capabilities. This enables the image denoising fusion model to perform more accurate denoising on images taken by different cameras under different exposure levels, thereby improving the overall quality of the fused image output by the image denoising fusion model.

[0296] Based on the same technical concept, embodiments of this application also provide a computer-readable storage medium storing a computer-executable program, which, when invoked by a computer, causes the computer to perform one or more steps in any of the above method embodiments.

[0297] Based on the same technical concept, embodiments of this application also provide a chip system, including a processor coupled to a memory, which executes a computer-executable program stored in the memory to implement one or more steps in any of the above method embodiments. This chip system can be a single chip or a chip module composed of multiple chips.

[0298] Based on the same technical concept, this application also provides a computer executable program product that, when run on an electronic device, causes the electronic device to perform one or more steps in any of the above method embodiments.

[0299] In the above embodiments, the descriptions of each embodiment have different focuses. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the sequence numbers of the steps in the above embodiments do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0300] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0301] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0302] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application. Therefore, the protection scope of the embodiments of this application should be determined by the protection scope of the claims.

Claims

1. A shooting method, characterized in that, include: Upon receiving a shooting instruction, and when the scene to be shot in the shooting preview frame is the target scene, at least one set of images is captured using multiple cameras; The target scene includes a fireworks scene; each set of images includes: multiple images with the same exposure time but different exposure amounts taken simultaneously by the multiple cameras of the scene to be photographed; in each set of images, the image with a smaller exposure amount is taken by the camera with better light sensitivity among the multiple cameras, and the image with a larger exposure amount is taken by the camera with poorer light sensitivity among the multiple cameras; the shooting time is different for different sets of images; Each group of images undergoes a first processing step to ensure that all images in each group have the same field of view, size, and resolution; each group of images processed by the first step constitutes a group of images to be fused. Multiple groups of images to be fused that are captured at adjacent times are input into a trained image denoising and fusion model for processing to obtain a fused image; The image denoising fusion model includes multiple denoising units, each trained on different training datasets. Each training dataset includes multiple sample data points, each containing multiple noisy images and one real image without noise. The noisy images are used as input to the denoising units during training. Correspondingly, the step of obtaining the noisy images in the generation of the training datasets for each of the multiple denoising units includes: copying multiple copies of the real image; performing image quality blurring on the copy image corresponding to the wide field-of-view camera; performing sensitivity compensation on the blurred copy image; adding a first noise to the sensitivity-compensated copy image based on the noise coefficient of the wide field-of-view camera; and converting the copy image with the added first noise into a raw format image to obtain the noisy image corresponding to the wide field-of-view camera.

2. The shooting method according to claim 1, characterized in that, The image denoising and fusion model further includes an image fusion unit; the total number of the plurality of denoising units is equal to the total number of the plurality of cameras, and the plurality of denoising units correspond one-to-one with the plurality of cameras; the denoising unit is used to fuse and denoise multiple images taken by the corresponding camera under the same exposure. Correspondingly, multiple groups of images to be fused that are captured at adjacent times are input into a trained image denoising and fusion model for processing to obtain a fused image, including: Each of the denoising units performs denoising fusion on multiple images captured by the camera corresponding to the denoising unit under the same exposure, and outputs a denoised image to the image fusion unit. The image fusion unit fuses multiple denoised images from the multiple denoising units under different exposure levels to obtain the fused image.

3. The shooting method according to claim 2, characterized in that, The training datasets corresponding to each of the multiple denoising units are generated in the following way: Acquire multiple high-resolution night scene images; For each high-definition night scene image, a region is randomly cropped from the high-definition night scene image as the image to be degraded; The image to be degraded is subjected to random preprocessing; The random preprocessing includes: random white balance processing, random rotation, random addition of ghosting, random addition of color blocks, and random addition of jitter blur; The image to be degraded after the random preprocessing is determined as a real image; The total number of the multiple duplicate images is equal to the total number of the multiple cameras; each of the multiple duplicate images corresponds one-to-one with one of the multiple cameras; A first noise-adding process is performed on the copy image corresponding to the large field-of-view camera among the plurality of cameras to obtain a noisy image corresponding to the large field-of-view camera. A second noise-adding process is applied to the copy image corresponding to the small field-of-view camera among the plurality of cameras to obtain a noisy image corresponding to the small field-of-view camera; the second noise-adding process is different from the first noise-adding process. The noisy image corresponding to the large field-of-view camera and the real image are used as one sample data of the denoising unit corresponding to the large field-of-view camera; the noisy image corresponding to the small field-of-view camera and the real image are used as one sample data of the denoising unit corresponding to the small field-of-view camera.

4. The shooting method according to claim 3, characterized in that, A second noise-adding process is performed on the copy image corresponding to the small field-of-view camera among the plurality of cameras to obtain a noisy image corresponding to the small field-of-view camera, including: The copy image corresponding to the small field-of-view camera is brightened; Based on the noise coefficient of the small field-of-view camera, a second noise is added to the brightened copy image; The copy image with added second noise is converted into a raw format image to obtain the noisy image corresponding to the small field-of-view camera.

5. The shooting method according to any one of claims 1-4, characterized in that, The first processing includes cropping, scaling, and upsampling; correspondingly, the first processing is performed on each group of images, including: The large field-of-view image in each group of images is cropped to obtain the cropped image corresponding to the large field-of-view image; the field of view corresponding to the cropped image is the same as the field of view corresponding to the small field-of-view image in the same group of images; the large field-of-view image refers to the image captured by the large field-of-view camera among the multiple cameras, and the small field-of-view image refers to the image captured by the small field-of-view camera among the multiple cameras; The cropped image corresponding to the large field-of-view image in each group of images is scaled to obtain the scaled image corresponding to the large field-of-view image; the size of the scaled image is the same as the size of the small field-of-view image in the same group of images, and the field of view corresponding to the scaled image is the same as the field of view corresponding to the small field-of-view image in the same group of images. Upsample the scaled image corresponding to the large field-of-view image in each group of images to obtain the upsampled image corresponding to the large field-of-view image; the resolution of the upsampled image is the same as the resolution of the small field-of-view image in the same group of images, and the size of the upsampled image is the same as the size of the small field-of-view image in the same group of images, and the field of view corresponding to the upsampled image is the same as the field of view corresponding to the small field-of-view image in the same group of images.

6. The shooting method according to any one of claims 1-4, characterized in that, The first processing further includes: distortion correction; correspondingly, the first processing is performed on each group of images, further including: Based on the calibration intrinsic parameters of the camera corresponding to each image in each group of images, distortion correction is performed on each image; the calibration intrinsic parameters include distortion coefficient and focal length.

7. The shooting method according to any one of claims 1-4, characterized in that, The first processing further includes: affine transformation; correspondingly, performing the first processing on each group of images further includes: Based on the calibration extrinsic parameters of the camera corresponding to each image in each group of images, an affine transformation is performed on each image; the calibration extrinsic parameters are used to describe the pose of the camera relative to the spatial coordinate system.

8. An electronic device, characterized in that, include: One or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 7.

10. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 7.

Images