Shooting method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a shooting method and device, equipment and a storage medium, and the method comprises the steps: obtaining a target image pair obtained through shooting in the same scene, the target image pair comprises a long-exposure image and a short-exposure image, and the exposure value of the long-exposure image is greater than the exposure value of the short-exposure image; performing alignment processing on the background region and the foreground region of the target image pair to obtain an aligned short exposure image and an aligned long exposure image; and performing fusion processing on the aligned short-exposure image and the aligned long-exposure image to obtain a shot image. The alignment accuracy between the long exposure image and the short exposure image can be improved, and then the imaging effect of the shot image is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to computer technology, and relate to but are not limited to a shooting method and apparatus, equipment, and storage medium. Background Art

[0002] Today, with the rapid development of digital photography technology, electronic devices are the most popular shooting tools in daily life, and their convenience and immediacy are deeply loved by users. However, during the shooting process of electronic devices, especially in complex lighting environments, it is often difficult to accurately control the exposure, resulting in the problem of overexposure or underexposure in the captured images.

[0003] To solve this problem, developing an efficient fusion shooting technology of long exposure images and short exposure images has become one of the keys to improving the shooting quality of electronic devices. Summary of the invention

[0004] In view of this, the shooting method, device, equipment, and storage medium provided in the embodiments of the present application can improve the alignment accuracy between the long exposure image and the short exposure image, thereby improving the imaging effect of the captured image. The shooting method, device, equipment, and storage medium provided in the embodiments of the present application are implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a shooting method, including:

[0006] Acquire a target image pair captured under the same scene, the target image pair comprising a long-exposure image and a short-exposure image, wherein an exposure value of the long-exposure image is greater than an exposure value of the short-exposure image;

[0007] Performing alignment processing on the background area and the foreground area of ​​the target image pair respectively to obtain an aligned short-exposure image and an aligned long-exposure image;

[0008] The aligned short-exposure image and the aligned long-exposure image are fused to obtain a captured image.

[0009] In a second aspect, an embodiment of the present application provides a photographing device, including:

[0010] An acquisition module, used to acquire a target image pair captured under the same scene, wherein the target image pair includes a long-exposure image and a short-exposure image, and an exposure value of the long-exposure image is greater than an exposure value of the short-exposure image;

[0011] An alignment module, used to align the background area and the foreground area of ​​the target image pair respectively to obtain an aligned short-exposure image and an aligned long-exposure image;

[0012] The fusion module is used to fuse the aligned short exposure image and the aligned long exposure image to obtain a captured image.

[0013] In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, the method described in the embodiment of the present application is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in the embodiment of the present application.

[0015] The shooting method, apparatus, computer device and computer-readable storage medium provided in the embodiments of the present application obtain a long exposure image and a short exposure image shot in the same scene, and align the long exposure image and the short exposure image in the background area and the foreground area respectively to obtain an aligned long exposure image and an aligned short exposure image, and then fuse the aligned long exposure image and the aligned short exposure image to obtain a shot image.

[0016] In this way, by aligning the foreground area and the background area of ​​the two images respectively, the alignment accuracy can be improved, and the alignment failure that occurs when the corresponding areas of the long exposure image and the short exposure image are directly aligned can be avoided. Distortion and deformation in the obtained captured image can also be avoided, thereby improving the imaging effect of the captured image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0018] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0019] Figure 2 A schematic diagram of an implementation flow of a photographing method provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of the implementation process of another shooting method provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of an implementation flow of aligning a background area of ​​a short exposure image and a background area of ​​a long exposure image to obtain an aligned short exposure image provided in an embodiment of the present application;

[0022] Figure 5A schematic diagram of the effects before and after optimization provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of the overall process of the photographing method provided in the embodiment of the present application;

[0024] Figure 7 A schematic diagram of the structure of a photographing device provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0028] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0029] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are used to distinguish similar or different objects, and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0030] Before introducing the relevant technical solutions of the present application, the terms involved in the embodiments of the present application are first explained.

[0031] Optical flow: The concept of optical flow was first proposed by Gibson in 1950. It is the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane. It is a method that uses the change of pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. Optical flow information is helpful for multi-frame noise reduction, video interpolation, fusion, etc.

[0032] Dense Inverse Search (DIS): The DIS algorithm is an algorithm for fast optical flow calculation. It efficiently calculates the optical flow field between images by densely searching for reverse matching points. It is a method for optical flow calculation that uses reverse search and gradient descent to seek optimization. Specifically, it is based on block-level correlation reverse search, which calculates the gradient once and then uses it for multiple reverse searches without reinitializing the gradient calculation each time.

[0033] Long exposure image (Exposure Value 0, Ev0): The camera takes a long exposure time, which may range from a few seconds to several minutes. Long exposure can capture more light and is suitable for low-light environments or scenes that require motion blur. Long exposure will cause dynamic elements in the image to blur because light continues to act on the photosensitive element during the exposure time. Long exposure images can capture more details and color levels in low-light environments.

[0034] Short exposure image (Exposure Value m, Evm): The exposure time of the camera is short, usually between a fraction of a second and several seconds. This exposure method can capture the picture quickly and reduce the blur or motion track caused by long exposure. Due to the short exposure time, the dynamic elements in the image (such as people, animals, vehicles, etc.) usually remain clear, and can accurately capture the instantaneous action and details. In addition, the color and contrast of short exposure images are usually more vivid, because the short exposure time reduces the overexposure caused by excessive light.

[0035] When using electronic devices for actual shooting, such as front-facing shooting, the lighting environment may be complex and changeable. In this way, the background area may easily be overexposed. In order to restore the overexposed background area information, the industry usually merges the long-exposure image with the short-exposure image, so that the final image can present better results in different lighting environments.

[0036] When the industry fuses long-exposure images with short-exposure images, the overexposed area in the long-exposure image is generally replaced with the corresponding area in the short-exposure image to minimize the overexposed area of ​​the image and ensure high dynamics.

[0037] However, when calculating the alignment of the short-exposure image to the long-exposure image based on the optical flow algorithm, slight distortion will occur in the overexposed area. Because there is no valid information in the overexposed area of ​​the long-exposure image, there will be deviations when calculating the optical flow. Based on the results calculated by the optical flow algorithm, due to local displacement in the image (such as inconsistent movement of the portrait in the foreground area and the background area), alignment failure will occur in most cases, thus affecting the presentation of the final captured image.

[0038] In view of this, an embodiment of the present application provides a shooting method, which is applied to an electronic device. The electronic device may include but is not limited to a mobile phone, a wearable device (such as a smart watch, a smart bracelet, smart glasses, etc.), a tablet computer, a laptop computer, a vehicle terminal, a PC (Personal Computer), etc. The function implemented by the method can be implemented by calling a program code by a processor in the electronic device. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device at least includes a processor and a storage medium.

[0039] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0040] For example, Figure 1 As shown, the electronic device 10 may include a processor 101 , an external memory interface 102 , an internal memory 103 and a universal serial bus (USB) interface 104 .

[0041] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 10. In other embodiments of the present application, the electronic device 10 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0042] The processor 101 may include one or more processing units, for example: the processor 101 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural network processing unit (neural network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. Exemplarily, the processor 101 can be a smart terminal CPU, such as a Snapdragon series processor, etc. In some embodiments, the processor 101 may include one or more interfaces. The interface may include an inter integrated circuit (I2C) interface, an inter integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0043] The external memory interface 102 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 10. The external memory card communicates with the processor 101 through the external memory interface 102 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0044] The internal memory 103 can be used to store computer executable program codes, which include instructions. The internal memory 103 can include a program storage area and a data storage area. The program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 10 (such as audio data, a phone book, etc.), etc.

[0045] In addition, the electronic device involved in the embodiments of the present application may also be installed with an operating system, and application programs may be installed and run on the operating system, which is not limited in the embodiments of the present application.

[0046] Figure 2 The following is a schematic diagram of the implementation process of the shooting method provided in the embodiment of the present application. Figure 2 As shown, the method may include the following steps 201 to 203:

[0047] Step 201 : acquiring a target image pair captured under the same scene, wherein the target image pair includes a long-exposure image and a short-exposure image, and an exposure value of the long-exposure image is greater than an exposure value of the short-exposure image.

[0048] In the embodiment of the present application, there is no limitation on the method of acquiring the target image pair captured in the same scene. For example, the target image pair may be two frames of images connected in time sequence, and when acquiring the two frames of images connected in time sequence, the exposure value of the electronic device is different.

[0049] Alternatively, the target image pair may also be two non-temporally connected frames of images captured under the same scene. It is only necessary to ensure that the exposure values ​​of the electronic device are different when acquiring the two non-temporally connected frames of images.

[0050] For example, an electronic device obtains four images, namely frame 1, frame 2, frame 3 and frame 4, taken in the same scene, where frame 1 is a long exposure image and frame 4 is a short exposure image. Then the obtained target image pair may be frame 1 and frame 4.

[0051] In one embodiment, in order to facilitate user operation, multiple shooting modes may be set in the electronic device, such as a professional mode, a streamer shutter mode, a special mode, etc. In the special mode, the electronic device can respond to a shutter operation and simultaneously capture two images with different exposure values. For example, in the special mode, the camera of the electronic device will automatically set different shutter speeds to obtain long-exposure images and short-exposure images respectively. For example, a short-exposure image may use a faster shutter speed (such as 1 / 100 seconds), while a long-exposure image uses a slower shutter speed (such as several seconds or even longer).

[0052] Step 202 , aligning the background area and the foreground area of ​​the target image pair respectively to obtain an aligned short-exposure image and an aligned long-exposure image.

[0053] Understandably, when shooting with an electronic device, if there is insufficient light, although the short-exposure image can capture the highlights in the scene and avoid overexposure, the dark areas of the face in the short-exposure image may lose details, which will affect the overall clarity and expression of the face. Although the long-exposure image taken by the electronic device can capture the dark details in the scene and avoid the unclearness caused by underexposure of the dark details, the background area in the long-exposure image may be overexposed, resulting in less available information in the background area.

[0054] Therefore, a method of fusing the long exposure image and the short exposure image may be selected so that the details of the dark area of ​​the final captured image are clearer and the highlight area is not overexposed.

[0055] Based on this, in an embodiment of the present application, the background area and the foreground area of ​​the target image pair may be aligned respectively to obtain an aligned short exposure image and an aligned long exposure image.

[0056] In some embodiments, in order to achieve alignment processing of the background area and the foreground area of ​​the target image pair, the background area of ​​the short exposure image and the background area of ​​the long exposure image can be selectively aligned to obtain an aligned short exposure image, so that the obtained aligned short exposure image contains useful information of the background area of ​​the short exposure image; then, the foreground area of ​​the long exposure image and the foreground area of ​​the aligned short exposure image are aligned to obtain an aligned long exposure image, which can avoid distortion in the obtained aligned long exposure image and improve the alignment accuracy of the foreground area (portrait area).

[0057] Step 203 , fusing the aligned short-exposure image and the aligned long-exposure image to obtain a captured image.

[0058] In the embodiment of the present application, after obtaining the aligned short exposure image and the aligned long exposure image, the two can be fused to obtain the final presented image, that is, to obtain the captured image.

[0059] In the embodiment of the present application, there is no limitation on the implementation method of fusing the aligned short exposure image and the aligned long exposure image. For example, a weighted average method, a pyramid fusion method, or a more complex fusion algorithm such as a multi-resolution fusion method or a gradient-based fusion method may be used.

[0060] In an embodiment of the present application, a long exposure image and a short exposure image captured in the same scene are acquired, and the long exposure image and the short exposure image are aligned in the background area and the foreground area respectively to obtain an aligned long exposure image and an aligned short exposure image, and then the aligned long exposure image and the aligned short exposure image are fused to obtain a captured image.

[0061] In this way, by aligning the foreground area and the background area of ​​the two images respectively, the alignment accuracy can be improved, and the alignment failure that occurs when the corresponding areas of the long exposure image and the short exposure image are directly aligned can be avoided. Distortion and deformation in the obtained captured image can also be avoided, thereby improving the imaging effect of the captured image.

[0062] Figure 3 The following is a schematic diagram of the implementation process of the shooting method provided in the embodiment of the present application. Figure 3 As shown, the method may include the following steps 301 to 308:

[0063] Step 301 , obtaining a target image pair captured under the same scene, wherein the target image pair includes a long-exposure image and a short-exposure image, and an exposure value of the long-exposure image is greater than an exposure value of the short-exposure image.

[0064] Here, the method of executing step 301 is the same as the method of executing step 201 in the above embodiment, and will not be repeated here.

[0065] Step 302 : segment the short exposure image to obtain a foreground area and a background area corresponding to the short exposure image.

[0066] In the embodiment of the present application, when the short exposure image is segmented, the segmentation method to be performed is not limited. For example, the segmentation method may include a segmentation method based on traditional image processing, such as threshold segmentation, that is, by setting one or more thresholds, the pixels of the image are divided into different categories to achieve segmentation; or a segmentation method including edge detection, using the edge information of the image to achieve segmentation by detecting the edge, etc.

[0067] The implementation method of segmentation processing may also include a segmentation method based on deep learning, such as semantic segmentation, which uses a deep learning model (such as a convolutional neural network) to classify images at the pixel level to achieve segmentation; or includes instance segmentation, which further distinguishes different instances (i.e., different objects of the same category) based on semantic segmentation; or includes a segmentation method for human posture estimation, which detects key points of the human body (such as joints, head, etc.) and constructs a skeleton model of the human body, thereby achieving portrait segmentation, etc.

[0068] After the short exposure image is segmented, the foreground area and background area corresponding to the short exposure image can be obtained. In some embodiments, if the shooting mode is front shooting, the foreground area corresponding to the short exposure image may include an object, such as a portrait, an animal, a tree, etc., without limitation.

[0069] In the following, the shooting mode of the electronic device is front shooting, and the foreground area corresponding to the short exposure image includes a portrait. After the short exposure image is segmented, a portrait segmentation mask (segmask) can be obtained. Each pixel value in the segmask indicates whether the pixel belongs to the portrait area, and is usually represented by binary value, 0 for the non-portrait area and 1 for the portrait area.

[0070] Step 303: Determine the global optical flow field from the short exposure image to the long exposure image according to a preset optical flow algorithm.

[0071] In the embodiments of the present application, there is no limitation on the specific type of the preset optical flow algorithm, such as the Lucas-Kanade algorithm, Horn-Schunck algorithm or DIS algorithm, etc. Among them, the Lucas-Kanade algorithm is an optical flow estimation algorithm based on the assumption of local brightness consistency, which estimates the movement speed of the object by solving the optical flow equations; the Horn-Schunck algorithm is an optical flow estimation algorithm based on the assumption of global smoothness and brightness constancy, which can generate dense optical flow field estimation, that is, the optical flow value is calculated for each pixel point, and the algorithm can still maintain a certain accuracy and robustness in complex scenes and large changes in illumination.

[0072] The DIS algorithm is a dense optical flow algorithm used to calculate the optical flow between two images. It mainly uses the intensity changes of pixels in the image to estimate the movement of pixels to obtain the optical flow field.

[0073] Here, the DIS algorithm is taken as an example to illustrate that in order to calculate the global optical flow field from the short exposure image to the long exposure image, the short exposure image and the long exposure image can be preprocessed first, such as converting the short exposure image and the long exposure image into grayscale images to reduce the subsequent calculation amount; then a pyramid is constructed for the two grayscale images, and each layer of the pyramid is obtained by downsampling the image of the previous layer, so that motion information can be captured at different scales.

[0074] Subsequently, optical flow calculation may be performed for the two images, which may include the following steps:

[0075] (1) Gradient field calculation: The gradient of the grayscale image is calculated to obtain the gradient field of the image in the x and y directions. This gradient information will be used for subsequent optical flow estimation.

[0076] (2) Sampling and window operation: For each pixel, the DIS optical flow algorithm uses a fixed-size window to sample around it and obtain a set of sampling points. These sampling points are all located in the neighborhood around the current pixel and are used to estimate the motion vector of the current pixel.

[0077] (3) Reverse search: The DIS optical flow algorithm uses a reverse search strategy to solve the optical flow. It first calculates the sparse optical flow field on the top-level pyramid image, and then uses these sparse optical flow fields to gradually derive the lower-level, denser optical flow fields. The core idea of ​​reverse search is to find the best matching position by comparing the similarity between image blocks, thereby estimating the optical flow.

[0078] (4) Densification: After obtaining the sparse optical flow field, the DIS optical flow algorithm densifies it using methods such as interpolation or weighted summation to obtain the optical flow vector corresponding to each pixel, and finally obtains the global optical flow field from the short exposure image to the long exposure image.

[0079] The DIS algorithm can be implemented in two steps: (1) Initialization: construct an image pyramid for two frames of images and perform some calculations for initialization; (2) Iteration: iterate from the top layer of the image pyramid to the bottom layer, and calculate the block integral image of the image based on the gradient image of the current layer; then reverse search to solve the sparse optical flow, calculate the dense optical flow based on the sparse optical flow, and obtain the final global optical flow field (expressed as an optical flow image).

[0080] After the global optical flow field from the short exposure image to the long exposure image is determined based on the preset optical flow algorithm, the global optical flow field describes the movement of all pixels in the image and can reflect the overall movement trend of the image.

[0081] Step 304 , aligning the background area of ​​the short exposure image and the background area of ​​the long exposure image according to the global optical flow field to obtain an aligned short exposure image.

[0082] In an embodiment of the present application, after calculating the global optical flow field from the short exposure image to the long exposure image, the background area of ​​the short exposure image and the background area of ​​the long exposure image can be further aligned in coordinate position according to the global optical flow field to obtain an aligned short exposure image (which can be called evmwarped). The aligned short exposure image retains the information of the foreground area in the short exposure image (such as portrait information) while being consistent with the background area of ​​the long exposure image.

[0083] In some embodiments, Figure 4As shown, a schematic diagram of a process of aligning the background area of ​​a short exposure image and the background area of ​​a long exposure image according to a global optical flow field to obtain an aligned short exposure image is provided, including the following steps 401 to 402:

[0084] Step 401 , extracting corner point features of the background region of the short exposure image, and extracting a background optical flow field corresponding to the background region of the short exposure image from the global optical flow field according to the corner point features of the background region of the short exposure image.

[0085] In the embodiment of the present application, there is no limitation on the implementation method of extracting the corner point features of the background area of ​​the short exposure image. For example, Harris corner point detection, Shi-Tomasi corner point detection, etc. can be used to find stable feature points (corner points) on the background area of ​​the short exposure image (such as segmented as the background image). Corner points are the points where the edge of the image changes the fastest, usually have significant features, and are easy to detect and track. In optical flow calculation, corner points are often used as feature points to estimate the motion of objects.

[0086] In an embodiment of the present application, after obtaining the corner point features corresponding to the background area of ​​the short exposure image, the background optical flow field corresponding to the background area of ​​the short exposure image can be extracted from the global optical flow field corresponding to the short exposure image based on the corner point features. The background optical flow field can accurately reflect the motion information of the background area of ​​the captured short exposure image.

[0087] Here, by using corner points as feature points, the background area of ​​the short exposure image can be effectively determined, and by using corner point features, the calculation of optical flow can be accelerated and the accuracy of optical flow can be improved.

[0088] Step 402 : aligning the background area of ​​the short exposure image and the background area of ​​the long exposure image according to the background optical flow field to obtain an aligned short exposure image.

[0089] In an embodiment of the present application, after obtaining the background optical flow field corresponding to the background area of ​​the short exposure image, the background optical flow field can also be applied to the global warp process of the image to transform the image to a new position or shape, so that the background area of ​​the short exposure image can be aligned in position with the background area of ​​the long exposure image, thereby eliminating image distortion and deformation caused by camera motion or object motion.

[0090] In some embodiments, in order to achieve position alignment processing of the background area of ​​the short exposure image and the background area of ​​the long exposure image, the global transformation matrix corresponding to the background area of ​​the short exposure image can be first determined based on the background optical flow field corresponding to the background area of ​​the short exposure image; then, the background area of ​​the short exposure image is aligned with the background area of ​​the long exposure image through the global transformation matrix to obtain an aligned short exposure image.

[0091] Exemplarily, the above method may include the following steps:

[0092] (1) Feature points can be detected in the background area of ​​the short exposure image and the background area of ​​the long exposure image respectively. For example, feature point detection algorithms such as SIFT, SURF, and ORB can be used to obtain corresponding feature points. The detected feature points are then matched to find corresponding feature point pairs in the background area of ​​the short exposure image and the background area of ​​the long exposure image. Here, algorithms such as Brute-Force Matcher or FLANN (Fast Library for Approximate Nearest Neighbors) matcher can be used to achieve feature point matching.

[0093] (2) Calculate the global transformation matrix: The matched feature point pairs can be used to estimate the global transformation matrix (such as an affine matrix). For example, the RANSAC (Random Sample Consensus) algorithm can be used to estimate the transformation matrix, which can handle the mismatch problem in the feature point pairs.

[0094] (3) Verify the transformation matrix: Use the estimated transformation matrix to transform the background area of ​​the short exposure image and check whether the transformed image is aligned with the background area of ​​the long exposure image. For example, the validity of the transformation matrix can be verified by calculating the alignment error or visual inspection. After verification, the global transformation matrix can be put into use.

[0095] (4) Applying a global transformation matrix: Use the estimated global transformation matrix to align the background region of the short exposure image to the background region of the long exposure image. This usually involves interpolating the image to ensure that the transformed image remains visually smooth and coherent.

[0096] By implementing the above embodiment, the position alignment of the background area of ​​the short exposure image to the background area of ​​the long exposure image can be achieved, so that the image distortion and deformation caused by camera movement or object movement is eliminated in the obtained aligned short exposure image.

[0097] Step 305 : determining the local optical flow field in the foreground area from the long exposure image to the aligned short exposure image according to a preset optical flow algorithm.

[0098] Here, the specific type of the preset optical flow algorithm is also not limited. For related introduction, please refer to step 303, which will not be repeated here.

[0099] In an embodiment of the present application, after acquiring the aligned short exposure image, the local optical flow field in the foreground area from the long exposure image to the aligned short exposure image can be further determined to obtain a motion vector of a specific area in the long exposure image (such as a portrait in the foreground area) relative to the aligned short exposure image.

[0100] Step 306 , aligning the foreground area of ​​the long exposure image and the foreground area of ​​the aligned short exposure image according to the local optical flow field corresponding to the long exposure image, to obtain the aligned long exposure image.

[0101] In some embodiments, after obtaining the local optical flow field corresponding to the long exposure image, the local transformation matrix corresponding to the foreground area of ​​the long exposure image can be determined according to the local optical flow field corresponding to the long exposure image; and then, the foreground area of ​​the long exposure image is aligned with the foreground area of ​​the aligned short exposure image through the local transformation matrix to obtain the aligned long exposure image.

[0102] Exemplarily, the above method may include the following steps:

[0103] (1) Feature points can be detected in the foreground area of ​​the long exposure image and the foreground area of ​​the aligned short exposure image respectively. For example, feature point detection algorithms such as SIFT, SURF, and ORB can be used to obtain corresponding feature points. The detected feature points are then matched to find corresponding feature point pairs in the foreground area of ​​the long exposure image and the foreground area of ​​the aligned short exposure image.

[0104] (2) Calculate the local transformation matrix: The matched feature point pairs can be used to estimate the local transformation matrix (such as an affine matrix). For example, the RANSAC (Random Sample Consensus) algorithm can be used to estimate the transformation matrix, which can handle the mismatch problem in the feature point pairs.

[0105] (3) Verify the transformation matrix: Use the estimated transformation matrix to transform the foreground area of ​​the long exposure image, and check whether the transformed image is aligned with the foreground area of ​​the aligned short exposure image. For example, the validity of the transformation matrix can be verified by calculating the alignment error or visual inspection. After verification, the local transformation matrix can be put into use.

[0106] (4) Applying the local transformation matrix: Use the estimated local transformation matrix to align the foreground area of ​​the long exposure image with the foreground area of ​​the aligned short exposure image. This usually involves interpolating the images to ensure that the transformed image remains visually smooth and coherent.

[0107] Here, the local optical flow information corresponding to the foreground area in the long exposure image is used to perform local warping on the long exposure image, so that the objects (such as portraits) in the foreground area in the long exposure image and the objects in the foreground area in the aligned short exposure image are kept consistent in position.

[0108] Step 307 , performing image alignment processing on the aligned short exposure image and the aligned long exposure image to obtain an alignment result, wherein the alignment result includes a region matching degree belonging to the same sub-region in the aligned short exposure image and the aligned long exposure image, and the sub-region includes multiple regions.

[0109] In the embodiment of the present application, the specific implementation method of performing image alignment processing on the aligned short exposure image and the aligned long exposure image is not limited.

[0110] For example, in some embodiments, to determine the alignment result of the aligned short exposure image and the aligned long exposure image, a confidence map (Confidence Map) between the aligned short exposure image and the aligned long exposure image may be calculated. The Confidence Map is an image representing the quality or reliability of the alignment. In the context of image alignment, it is usually used to indicate which areas are successfully aligned (i.e., high confidence) and which areas may have problems (i.e., low confidence).

[0111] Here, there is no limitation on the method of calculating the confidence map between the aligned short exposure image and the aligned long exposure image. For example, the confidence map may be calculated based on a variety of factors, such as the number of feature matches, alignment error, image gradient, etc.

[0112] Here, the confidence map can be used to characterize the regional matching degree of the same sub-region in the aligned short exposure image and the aligned long exposure image. A high confidence region generally means that the alignment is accurate and reliable, that is, the regional matching degree is high, while a low confidence region may contain incorrectly aligned or misaligned parts, that is, the regional matching degree is low.

[0113] Step 308 : According to the alignment result, the aligned short exposure image and the aligned long exposure image are fused to obtain a captured image.

[0114] In an embodiment of the present application, after calculating the regional matching degrees belonging to the same sub-region in the aligned short exposure image and the aligned long exposure image, the aligned short exposure image and the aligned long exposure image can be fused based on the regional matching degrees of multiple sub-regions to obtain a captured image.

[0115] For example, in some embodiments, when the area matching degree of the target sub-region is greater than a matching degree threshold, the target sub-region in the aligned short exposure image may be replaced with the target sub-region in the aligned long exposure image to obtain a replaced short exposure image, where the target sub-region is one of multiple sub-regions; and the replaced short exposure image and the long exposure image are fused to obtain a captured image.

[0116] That is, once the confidence map determines that the alignment of a certain target sub-region in the aligned short exposure image and the aligned long exposure image is successful, that is, the region matching degree of a certain target sub-region is greater than the matching degree threshold, the target sub-region can be extracted from the aligned long exposure image and pasted to the corresponding position of the aligned short exposure image. Repeat the above process to obtain the replaced short exposure image.

[0117] In the process of obtaining the replaced short exposure image, in some embodiments, a smooth transition can be made between the replacement area and the surrounding area to avoid obvious stitching marks, thereby ensuring the smoothness of the replaced short exposure image and improving the presentation effect.

[0118] Furthermore, after obtaining the replaced short exposure image, the replaced short exposure image and the long exposure image may be further fused to obtain the captured image.

[0119] Here, there is no limitation on the specific implementation method of fusing the replaced short-exposure and long-exposure images. For example, weighted averaging, Laplace pyramid fusion, multi-resolution fusion and other technologies may be used for fusion to obtain the captured image.

[0120] In the embodiment of the present application, by fusing the aligned short-exposure image and the aligned long-exposure image, it is possible to ensure that the non-overexposed areas in the foreground area (portrait area) and the background area of ​​the fused captured image use the information of the long-exposure image to the maximum extent, and ensure that the image will not cause blur and noise problems due to the fusion of the information of the portrait area of ​​the short-exposure image. That is, the captured image retains both the dynamic information in the short-exposure image and the structure and details in the long-exposure image.

[0121] Figure 5 A schematic diagram of the effect before and after optimization is given.

[0122] like Figure 5As shown, in the embodiment of the present application, on the one hand, by segmenting the short exposure image, the foreground area and background area corresponding to the short exposure image are obtained, and then according to the preset optical flow algorithm, the global optical flow field from the short exposure image to the long exposure image is determined, and the corner point features of the background area of ​​the short exposure image are extracted. According to the corner point features of the background area of ​​the short exposure image, the background optical flow field corresponding to the background area of ​​the short exposure image is extracted from the global optical flow field, so that according to the background optical flow field corresponding to the background area, the background area of ​​the short exposure image is globally transformed to obtain the aligned short exposure image, which can ensure that the aligned short exposure image can align the over-exposed area while avoiding the problem of distortion caused by the lack of information in the over-exposed area of ​​the long exposure image. On the other hand, by determining the confidence map between the aligned short exposure image and the aligned long exposure image, and according to the confidence map, the information of the long exposure image to the short exposure image is restored, which can ensure that the non-over-exposed area in the foreground area (portrait area) and the background area of ​​the fused captured image uses the information of the long exposure image to the greatest extent, and ensure that the image will not cause blur and noise problems due to the fusion of the information of the portrait area of ​​the short exposure image.

[0123] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0124] Figure 6 The overall flow chart of the shooting method provided in the embodiment of the present application is as follows. Figure 6 As shown, when executing this method, the global warp image evmwarped (i.e., the aligned short exposure image) from the short exposure image to the long exposure image and the local transformation image ev0Tevmwarped (i.e., the aligned long exposure image) from the long exposure image to evmwarped (i.e., the aligned short exposure image) are calculated; then the Confidence Map of the warp between the evmwarped image and the ev0Tevmwarped image is calculated. The two images are fused under the guidance of the Confidence Map to obtain the final captured image.

[0125] Specifically, the following steps may be included:

[0126] (1) Calculate the portrait segmentation image Segmask (i.e., the foreground area) in the short exposure image so that the influence of this area can be excluded in subsequent processing. Here, a portrait segmentation algorithm (such as a deep learning model) can be used to process the short exposure image to obtain a portrait segmentation image.

[0127] (2) Calculate the corner features of the short exposure image, such as using a corner detection algorithm (such as Shi-Tomasi corner detection, Harris corner detection, etc.) to detect corner features in the short exposure image, and then remove the corner features of the portrait area from the corner features of the short exposure image by determining the portrait segmentation image (i.e., the foreground area) in (1), and only retain the corner features of the background area of ​​the short exposure image.

[0128] Then, the DIS algorithm is used to calculate the global optical flow field (global optical flow information) from the short exposure image to the long exposure image. The global optical flow of the background area of ​​the short exposure image is calculated through the corner point features and the global optical flow field of the background area of ​​the short exposure image, and the alignment result of the long exposure image to the short exposure image in the background area is obtained, that is, evmwarped (aligned short exposure image) is obtained.

[0129] In the calculation process, corner points can be used as feature points to speed up the calculation of optical flow and improve the accuracy of optical flow. Corner points are easy to establish corresponding relationships between adjacent frames because of their significant features.

[0130] After obtaining the Global optical flow field of the background area, it can be applied to the global warp process. This usually involves integrating the optical flow field or applying some transformation function (such as affine transformation, perspective transformation, etc.) to transform the image to a new position or shape.

[0131] The purpose of global warp is to align the background area in the image with the reference image or template, thereby eliminating the image distortion and deformation caused by camera motion or object motion. Global warp refers to the global transformation of an image so that it is aligned with another image in the coordinate system. In image processing, global warp is often used for tasks such as image registration and image fusion.

[0132] (3) The local optical flow field from the long exposure image to evmwarped (the aligned short exposure image) is calculated through the DIS algorithm, and the long exposure image is localwarped to obtain ev0Tevmwarped (i.e. the aligned long exposure image).

[0133] (4) The Confidence Map between ev0Tevmwarped (aligned long exposure image) and evmwarped (aligned short exposure image) is calculated, and the successfully aligned area is designated to return the long exposure image to the short exposure image (i.e., the area of ​​the long exposure image is replaced with the corresponding area of ​​the short exposure image), so as to obtain the replaced short exposure image.

[0134] (5) The replaced short exposure image and the long exposure image (original image) are fused to obtain the captured image.

[0135] In the embodiment of the present application, in the process of front-facing shooting using an electronic device, the non-portrait area of ​​the short-exposure image is subjected to global warp processing to ensure that the long-exposure image and the short-exposure image on the overexposed area are aligned. The long-exposure image is warped to the short-exposure image warped image, and then the short-exposure image is posted back according to the Confidence Map, so that the non-overexposed area can use the information of the long-exposure image as much as possible.

[0136] It should be understood that, although the steps in the above-mentioned flowcharts are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flowcharts may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0137] Based on the foregoing embodiments, an embodiment of the present application provides a shooting device, which includes the modules included and the units included in the modules, which can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0138] Figure 7 A schematic diagram of the structure of a photographing device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device 700 includes an acquisition module 701, an alignment module 702 and a fusion module 703, wherein:

[0139] An acquisition module 701 is used to acquire a target image pair captured under the same scene, wherein the target image pair includes a long-exposure image and a short-exposure image, and an exposure value of the long-exposure image is greater than an exposure value of the short-exposure image;

[0140] An alignment module 702 is used to align the background area and the foreground area of ​​the target image pair to obtain an aligned short-exposure image and an aligned long-exposure image;

[0141] The fusion module 703 is used to fuse the aligned short exposure image and the aligned long exposure image to obtain a captured image.

[0142] In some embodiments, the alignment module 702 includes a first alignment module and a second alignment module;

[0143] The first alignment module is used to align the background area of ​​the short exposure image and the background area of ​​the long exposure image to obtain the aligned short exposure image;

[0144] The second alignment module is used to perform alignment processing on the foreground area of ​​the long exposure image and the foreground area of ​​the aligned short exposure image to obtain the aligned long exposure image.

[0145] In some embodiments, the first alignment module includes a first determination submodule and a first alignment submodule;

[0146] The first determination submodule is used to determine the global optical flow field from the short exposure image to the long exposure image according to a preset optical flow algorithm;

[0147] The first alignment submodule is used to align the background area of ​​the short exposure image and the background area of ​​the long exposure image according to the global optical flow field to obtain the aligned short exposure image.

[0148] In some embodiments, the apparatus further comprises a segmentation module;

[0149] The segmentation module is used to perform segmentation processing on the short exposure image to obtain a foreground area and a background area corresponding to the short exposure image;

[0150] The first alignment submodule is further used to extract corner point features of the background area of ​​the short exposure image; extract a background optical flow field corresponding to the background area of ​​the short exposure image from the global optical flow field according to the corner point features of the background area of ​​the short exposure image; and align the background area of ​​the short exposure image and the background area of ​​the long exposure image according to the background optical flow field to obtain the aligned short exposure image.

[0151] In some embodiments, the first alignment submodule is specifically used to determine the global transformation matrix corresponding to the background area of ​​the short exposure image based on the background optical flow field corresponding to the background area of ​​the short exposure image; through the global transformation matrix, the background area of ​​the short exposure image is aligned with the background area of ​​the long exposure image to obtain the aligned short exposure image.

[0152] In some embodiments, the second alignment module includes a second determination submodule and a second alignment submodule.

[0153] The second determination submodule is used to determine the local optical flow field in the foreground area from the long exposure image to the aligned short exposure image according to a preset optical flow algorithm;

[0154] The second alignment submodule is used to align the foreground area of ​​the long exposure image and the foreground area of ​​the aligned short exposure image according to the local optical flow field corresponding to the long exposure image, so as to obtain the aligned long exposure image.

[0155] In some embodiments, the second alignment submodule is specifically configured to determine a local transformation matrix corresponding to a foreground area of ​​the long exposure image according to a local optical flow field corresponding to the long exposure image;

[0156] The foreground area of ​​the long exposure image is aligned with the foreground area of ​​the aligned short exposure image by using the local transformation matrix to obtain the aligned long exposure image.

[0157] In some embodiments, the fusion module 703 further includes a third alignment submodule and a fusion submodule;

[0158] The third alignment submodule is used to perform image alignment processing on the aligned short exposure image and the aligned long exposure image to obtain an alignment result, wherein the alignment result includes a region matching degree belonging to the same subregion in the aligned short exposure image and the aligned long exposure image, and the subregion includes multiple regions;

[0159] The fusion submodule is used to perform fusion processing on the aligned short exposure image and the aligned long exposure image according to the alignment result to obtain the captured image.

[0160] In some embodiments, the fusion submodule is specifically used to replace the target subregion in the aligned short exposure image with the target subregion in the aligned long exposure image when the region matching degree of the target subregion is greater than a matching degree threshold, so as to obtain a replaced short exposure image, wherein the target subregion is one of multiple subregions; and fuse the replaced short exposure image and the long exposure image to obtain a captured image.

[0161] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0162] It should be noted that in the embodiments of this application Figure 7 The division of modules in the shooting device shown is schematic and is only a logical functional division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0163] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or partly embodied in the form of a software product that contributes to the relevant technology. The computer software product is stored in a storage medium, including several instructions to enable an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0164] The embodiment of the present application provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0165] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0166] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0167] Those skilled in the art will understand that Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0168] In one embodiment, the photographing device provided by the present application can be implemented in the form of a computer program. The computer program can be Figure 8 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the above-mentioned device. The computer program composed of various program modules enables the processor to execute the steps in the method of each embodiment of the present application described in this specification.

[0169] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium, storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0170] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0171] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0172] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0173] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0174] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0175] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0176] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0177] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0178] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0179] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0180] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0181] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A shooting method, characterized in that: The method comprises: Acquire a target image pair captured under the same scene, the target image pair comprising a long-exposure image and a short-exposure image, wherein an exposure value of the long-exposure image is greater than an exposure value of the short-exposure image; Performing alignment processing on the background area and the foreground area of ​​the target image pair respectively to obtain an aligned short-exposure image and an aligned long-exposure image; The aligned short-exposure image and the aligned long-exposure image are fused to obtain a captured image.

2. The method according to claim 1, characterized in that The step of respectively aligning the background area and the foreground area of ​​the target image pair to obtain an aligned short-exposure image and an aligned long-exposure image includes: Performing an alignment process on the background area of ​​the short exposure image and the background area of ​​the long exposure image to obtain the aligned short exposure image; An alignment process is performed on the foreground area of ​​the long exposure image and the foreground area of ​​the aligned short exposure image to obtain the aligned long exposure image.

3. The method according to claim 1, characterized in that The aligning the background area of ​​the short exposure image and the background area of ​​the long exposure image to obtain the aligned short exposure image includes: Determining a global optical flow field from the short exposure image to the long exposure image according to a preset optical flow algorithm; According to the global optical flow field, the background area of ​​the short exposure image and the background area of ​​the long exposure image are aligned to obtain the aligned short exposure image.

4. The method according to claim 3, characterized in that Before performing global optical flow processing on the background area of ​​the short exposure image, the method further includes: Performing segmentation processing on the short exposure image to obtain a foreground area and a background area corresponding to the short exposure image; The step of aligning the background area of ​​the short exposure image and the background area of ​​the long exposure image according to the global optical flow field to obtain the aligned short exposure image includes: Extracting corner features of a background area of ​​the short exposure image; extracting a background optical flow field corresponding to the background area of ​​the short exposure image from the global optical flow field according to corner point features of the background area of ​​the short exposure image; According to the background optical flow field, the background area of ​​the short exposure image and the background area of ​​the long exposure image are aligned to obtain the aligned short exposure image.

5. The method according to claim 4, characterized in that The step of aligning the background area of ​​the short exposure image and the background area of ​​the long exposure image according to the background optical flow field to obtain the aligned short exposure image includes: Determining a global transformation matrix corresponding to the background area of ​​the short exposure image according to the background optical flow field corresponding to the background area of ​​the short exposure image; The global transformation matrix is ​​used to align the background area of ​​the short exposure image with the background area of ​​the long exposure image to obtain the aligned short exposure image.

6. The method according to claim 2, characterized in that The aligning the foreground area of ​​the long exposure image and the foreground area of ​​the aligned short exposure image to obtain the aligned long exposure image includes: Determine, according to a preset optical flow algorithm, a local optical flow field in a foreground area from the long exposure image to the aligned short exposure image; According to the local optical flow field corresponding to the long exposure image, the foreground area of ​​the long exposure image and the foreground area of ​​the aligned short exposure image are aligned to obtain the aligned long exposure image.

7. The method according to claim 6, characterized in that The step of aligning the foreground area of ​​the long exposure image and the foreground area of ​​the aligned short exposure image according to the local optical flow field corresponding to the long exposure image to obtain the aligned long exposure image includes: Determining a local transformation matrix corresponding to a foreground area of ​​the long exposure image according to a local optical flow field corresponding to the long exposure image; The foreground area of ​​the long exposure image is aligned with the foreground area of ​​the aligned short exposure image by using the local transformation matrix to obtain the aligned long exposure image.

8. The method according to any one of claims 1 to 7, characterized in that: The step of fusing the aligned short exposure image and the aligned long exposure image to obtain a captured image includes: Performing image alignment processing on the aligned short exposure image and the aligned long exposure image to obtain an alignment result, wherein the alignment result includes a region matching degree belonging to the same sub-region in the aligned short exposure image and the aligned long exposure image, wherein the sub-region includes a plurality of sub-regions; According to the alignment result, the aligned short exposure image and the aligned long exposure image are fused to obtain the captured image.

9. The method according to claim 8, characterized in that The step of fusing the aligned short-exposure image and the aligned long-exposure image according to the alignment result to obtain the captured image comprises: When the area matching degree of the target sub-region is greater than a matching degree threshold, replacing the target sub-region in the aligned short-exposure image with the target sub-region in the aligned long-exposure image to obtain a replaced short-exposure image, wherein the target sub-region is one of the multiple sub-regions; The replaced short-exposure image and the long-exposure image are fused to obtain a captured image.

10. A photographing device, characterized in that: include: An acquisition module, used to acquire a target image pair captured under the same scene, wherein the target image pair includes a long-exposure image and a short-exposure image, and an exposure value of the long-exposure image is greater than an exposure value of the short-exposure image; An alignment module, used to align the background area and the foreground area of ​​the target image pair respectively to obtain an aligned short-exposure image and an aligned long-exposure image; The fusion module is used to fuse the aligned short exposure image and the aligned long exposure image to obtain a captured image.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images