Snapshot image generation method, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides a snapshot image generation method, electronic equipment, a storage medium and a program product, the method is applied to the electronic equipment, and the method comprises the following steps: in a blurring mode, storing an image frame recently collected by a camera in a cache; acquiring a snapshot instruction; determining a key frame from the cache in response to the snapshot instruction; when the depth information of the key frame cannot be obtained according to a preset method, reconstructing the depth information of the key frame according to the image stored in the cache; and generating a blurred snapshot image according to the key frame and the depth information of the key frame. By adopting the embodiment, when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame can be reconstructed according to the image frame stored in the cache, the electronic equipment can obtain the depth information with better quality, and the display effect of generating a blurred snapshot image during snapshot can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for generating a captured image, an electronic device, a storage medium and a program product. Background Art

[0002] With the rapid development of electronic technology, mobile phones and other electronic devices are increasingly widely used, and users have higher and higher requirements for images obtained by using electronic devices. For example, when users capture images of wonderful moments, portraits or close-up photos, they hope that the captured images can present a blur effect. In the prior art, the blur effect of the captured images is not ideal.

[0003] Therefore, how to improve the blurring effect of captured images becomes a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a method for generating a captured image, an electronic device, a storage medium and a program product, which can improve the blurring effect of the captured image.

[0005] In a first aspect, the present application provides a method for generating a captured image, which is applied to an electronic device, the method comprising: in a blur mode, saving the image frame most recently captured by a camera into a cache; obtaining a snapshot instruction; in response to the snapshot instruction, determining a key frame from the cache; when the depth information of the key frame cannot be obtained according to a preset method, reconstructing the depth information of the key frame according to the image saved in the cache; and generating a blurred captured image according to the key frame and the depth information of the key frame.

[0006] By adopting this embodiment, when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame can be reconstructed according to the image frame saved in the cache. Through this embodiment, the electronic device can obtain depth information of better quality, which is beneficial to improving the display effect of the blurred captured image generated during the capture.

[0007] In some possible embodiments, the key frame may be the clearest image frame in the buffer.

[0008] In some possible embodiments, obtaining the snapshot instruction may include: obtaining the snapshot instruction triggered by the user; or, obtaining the snapshot instruction may include: obtaining the snapshot instruction automatically triggered by the electronic device when an image matching a preset action or posture is detected.

[0009] In some possible embodiments, the camera of the electronic device includes: a first camera and a second camera, and the key frame is an image captured by the first camera.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the preset method includes: when the image captured by the second camera synchronized with the key frame is saved in the cache, determining that the depth information of the key frame can be obtained; when the image captured by the second camera synchronized with the key frame is not saved in the cache, determining that the depth information of the key frame cannot be obtained.

[0011] This embodiment defines the preset method. When the depth information of the key frame can be obtained according to the preset method, the depth information is determined by the image captured by the second camera synchronized with the key frame and stored in the cache. When the image captured by the second camera synchronized with the key frame is not stored in the cache, it is determined that the depth information of the key frame cannot be obtained according to the preset method. In this case, the depth information of the key frame is reconstructed according to the image frame stored in the cache.

[0012] In combination with the first aspect, in certain implementations of the first aspect, reconstructing the depth information of the key frame based on the image stored in the cache includes: reconstructing the depth information of the key frame based on one or more pairs of images in the synchronized image pairs in the cache.

[0013] In combination with the first aspect, in certain implementations of the first aspect, reconstructing the depth information of the key frame based on one or more pairs of images in the synchronized image pair in the cache includes: determining the depth information corresponding to the one or more pairs of images respectively; determining the motion estimation information of the one or more pairs of images relative to the key frame; and determining the depth information of the key frame based on the depth information and motion estimation information corresponding to the one or more pairs of images respectively.

[0014] This embodiment defines how to use one or more pairs of images in the synchronized image pair in the cache to reconstruct the depth information of the key frame. Specifically, the depth information of the key frame is determined according to the depth information corresponding to the synchronized image pair in the cache and the motion estimation information.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the depth information of the key frame reconstructed based on the image stored in the cache includes: determining contour information of the target object based on the key frame; determining confidence information corresponding to the one or more pairs of images; determining depth information corresponding to the one or more pairs of images; determining motion estimation information of the one or more pairs of images relative to the key frame; and determining the depth information of the key frame based on the contour information, and the confidence information, depth information and motion estimation information corresponding to the one or more pairs of images.

[0016] This embodiment defines how to use one or more pairs of images in the synchronized image pair in the cache to reconstruct the depth information of the key frame. Specifically, the depth information of the key frame is determined based on the confidence information, depth information and motion estimation information corresponding to the synchronized image pair in the cache, and the contour information of the target object determined based on the key frame.

[0017] In combination with the first aspect, in certain implementations of the first aspect, the one or more pairs of images are the preceding images of the key frame; or, the one or more pairs of images are the succeeding images of the key frame; or, the multiple pairs of images are the preceding images and succeeding images of the key frame.

[0018] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute any one of the steps in the first aspect.

[0019] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the methods described in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product comprises executable instructions, and when the executable instructions are executed on a computer, the computer executes any one of the methods described in the first aspect.

[0021] It should be understood that the technical solutions provided in the second to fourth aspects above and their technical features can all correspond to the method for generating captured images provided in the first aspect and its possible implementation methods, so the beneficial effects that can be achieved are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a schematic diagram of a blurred snapshot image generated by an embodiment of the present application;

[0023] Figure 2 It is a flowchart of a method for generating a captured image provided in an embodiment of the present application;

[0024] Figure 3 It is a flowchart of a method for generating a captured image provided in an embodiment of the present application;

[0025] Figure 4 It is a flowchart of a method for generating a captured image provided in an embodiment of the present application;

[0026] Figure 5A It is a schematic diagram of data stored in a cache and key frames in an embodiment of the present application;

[0027] Figure 5B It is a schematic diagram of generating data stored in a cache and depth information corresponding to a key frame in an embodiment of the present application;

[0028] Fig. 6A It is a schematic diagram of data stored in a cache and key frames in an embodiment of the present application;

[0029] Figure 6B It is a schematic diagram of generating data stored in a cache and depth information corresponding to a key frame in an embodiment of the present application;

[0030] Figure 7 A schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0031] Figure 8 A software structure block diagram of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0033] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first instruction and the second instruction are to distinguish different user instructions, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different.

[0034] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0035] The method for generating a captured image provided in the embodiments of the present application can be applied to mobile phones, tablet computers, personal computers (PCs), personal digital assistants (PDAs), smart watches, netbooks, wearable electronic devices, augmented reality (AR) devices, virtual reality (VR) devices and other terminals with a capture function. It should be noted that the present application does not limit the specific type of electronic device. For simplicity, the electronic device in the embodiments of the present application is described by taking a mobile phone as an example.

[0036] The idea of ​​the present application includes: in blur mode, saving the image frame most recently captured by the camera to a cache; triggering the generation of a snapshot instruction when a user's triggered snapshot instruction is obtained, or when a wonderful moment matching a pre-configured action or posture is detected (for example, when the user jumps up); determining a key frame from the cache in response to the snapshot instruction; when the depth information of the key frame cannot be obtained according to a preset method, reconstructing the depth information of the key frame according to the image saved in the cache; generating a blurred snapshot image according to the key frame and the depth information of the key frame, such as Figure 1 It is a snapshot image with a blurred background generated by the snapshot image generation method provided in an embodiment of the present application after detecting that a user jumps up.

[0037] like Figure 2 As shown, in one embodiment of the present application, the method for generating a captured image includes the following steps: 201 to 205.

[0038] 201. In blur mode, save the image frame most recently captured by the camera into the cache.

[0039] The cache may be a circular queue, and multiple frames of images may be cached in the circular queue. In some possible implementations, the camera of the electronic device may include: a first camera and a second camera, and the key frame is an image captured by the first camera.

[0040] 202. Get the snapshot command.

[0041] The capture instruction may be generated based on the user's operation trigger, or may be generated when a wonderful moment matching a pre-configured action or posture is detected (such as when the user jumps up).

[0042] 203. In response to the snapshot instruction, determine the key frame from the cache.

[0043] In some possible implementations, a picture in the cache is automatically selected as a key frame.

[0044] 204. When the depth information of the key frame cannot be obtained according to a preset method, reconstruct the depth information of the key frame according to the image stored in the cache.

[0045] In some possible implementations, when an image captured by a second camera synchronized with a key frame is saved in a cache, it is determined that the depth information of the key frame can be obtained; when an image captured by a second camera synchronized with a key frame is not saved in the cache, it is determined that the depth information of the key frame cannot be obtained.

[0046] The cache does not store the image captured by the second camera synchronized with the key frame, which may include the following two situations: the first situation is that the cache does not store the image corresponding to the key frame (such as Figure 5A As shown, the main camera is the first camera, and the auxiliary camera is the second camera. If the fourth frame of the main camera stored in the cache is determined to be a key frame, the auxiliary camera skips the frame sampling and stores it in the cache relative to the main camera, that is, the first frame, the third frame, and the fifth frame of the auxiliary camera are stored in the cache. Since the auxiliary camera only stores the first frame, the third frame, and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. Figure 5A It can be seen that there are images in the cache that are synchronized with the first frame, the third frame, and the fifth frame of the main camera). The second situation is that the image corresponding to the key frame is saved in the cache, but the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is greater than the preset value. For example, if the preset value is 3 milliseconds, when the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 4 milliseconds, it is determined that the depth information of the key frame cannot be obtained according to the preset method. When the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 2 milliseconds, it is determined that the depth information of the key frame can be obtained according to the preset method. For the second situation, see Fig. 6A As shown in , the cache stores the image of the auxiliary camera corresponding to each frame of the image saved by the main camera, and the time difference between the first frame of this camera and the first frame of the auxiliary camera is less than the preset value, then the first frame of the main camera is synchronized with the first frame of the auxiliary camera. The time differences between the second frame, third frame, fourth frame and fifth frame of the main camera and the second frame, third frame, fourth frame and fifth frame of the auxiliary camera are respectively greater than the preset value, then Fig. 6A In the case of the above, it is determined that the image captured by the second camera synchronized with the key frame is not saved in the cache).

[0047] In some possible implementations, the depth information of the key frame is reconstructed based on one or more pairs of images in the synchronized image pair in the cache. Specifically, the one or more pairs of images may be the preceding images of the key frame; or, the one or more pairs of images may be the succeeding images of the key frame; or, the multiple pairs of images may be the preceding images and succeeding images of the key frame. For example, for Figure 5A In the scene shown, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed based on the nearest forward synchronization frame (the image pair corresponding to the fourth frame) and the subsequent synchronization frame (the image pair corresponding to the fifth frame) of the fourth frame in the cache. Fig. 6A In the scene shown, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed according to the forward synchronization frame of the fourth frame in the cache (the image pair corresponding to the first frame).

[0048] 205. Generate a blurred captured image according to the key frame and the depth information of the key frame.

[0049] By adopting this embodiment, when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame can be reconstructed according to the image frame saved in the cache. Through this embodiment, the electronic device can obtain depth information of better quality, which is beneficial to improving the display effect of the blurred captured image generated during the capture.

[0050] like Figure 3 As shown, in one embodiment of the present application, the method for generating a captured image includes the following steps: 301 to 307.

[0051] 301. In blur mode, the image frame most recently captured by the camera is saved in the cache.

[0052] In some possible implementations, the camera of the electronic device may include: a first camera and a second camera, and the key frame is an image captured by the first camera.

[0053] 302. Get the snapshot instruction.

[0054] The capture instruction may be generated based on the user's operation trigger, or may be generated when a wonderful moment matching a pre-configured action or posture is detected (such as when the user jumps up).

[0055] 303. In response to the snapshot instruction, determine the key frame from the cache.

[0056] In some possible implementations, a picture in the cache is automatically selected as a key frame.

[0057] 304. When the depth information of the key frame cannot be obtained according to the preset method, reconstruct the depth information of the key frame according to the image stored in the cache.

[0058] In some possible implementations, when an image captured by a second camera synchronized with a key frame is saved in a cache, it is determined that the depth information of the key frame can be obtained; when an image captured by a second camera synchronized with a key frame is not saved in the cache, it is determined that the depth information of the key frame cannot be obtained.

[0059] The cache does not store the image captured by the second camera synchronized with the key frame, which may include the following two situations: the first situation is that the cache does not store the image corresponding to the key frame (such as Figure 5A As shown, the main camera is the first camera, and the auxiliary camera is the second camera. If the fourth frame of the main camera stored in the cache is determined to be a key frame, the auxiliary camera skips the frame sampling and stores it in the cache relative to the main camera, that is, the first frame, the third frame, and the fifth frame of the auxiliary camera are stored in the cache. Since the auxiliary camera only stores the first frame, the third frame, and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. Figure 5A It can be seen that there are images in the cache that are synchronized with the first frame, the third frame, and the fifth frame of the main camera). The second situation is that the image corresponding to the key frame is saved in the cache, but the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is greater than the preset value. For example, if the preset value is 3 milliseconds, when the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 4 milliseconds, it is determined that the depth information of the key frame cannot be obtained according to the preset method. When the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 2 milliseconds, it is determined that the depth information of the key frame can be obtained according to the preset method. For the second situation, see Fig. 6A As shown in , the cache stores the image of the auxiliary camera corresponding to each frame of the image saved by the main camera, and the time difference between the first frame of this camera and the first frame of the auxiliary camera is less than the preset value, then the first frame of the main camera is synchronized with the first frame of the auxiliary camera. The time differences between the second frame, third frame, fourth frame and fifth frame of the main camera and the second frame, third frame, fourth frame and fifth frame of the auxiliary camera are respectively greater than the preset value, then Fig. 6A In the case of the above, it is determined that the image captured by the second camera synchronized with the key frame is not saved in the cache).

[0060] 305. Determine motion estimation information of the one or more pairs of images relative to the key frame.

[0061] In some possible implementations, the depth information of the key frame is reconstructed based on one or more pairs of images in the synchronized image pair in the cache. Specifically, the one or more pairs of images may be the preceding images of the key frame; or, the one or more pairs of images may be the succeeding images of the key frame; or, the multiple pairs of images may be the preceding images and succeeding images of the key frame. For example, for Figure 5A In the scene shown, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed based on the nearest forward synchronization frame (the image pair corresponding to the fourth frame) and the subsequent synchronization frame (the image pair corresponding to the fifth frame) of the fourth frame in the cache. Fig. 6A In the scene shown, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed according to the forward synchronization frame of the fourth frame in the cache (the image pair corresponding to the first frame).

[0062] 306. Determine the depth information of the key frame according to the depth information and motion estimation information corresponding to one or more pairs of images.

[0063] 307. Generate a blurred snapshot image based on the key frame and the depth information of the key frame.

[0064] By adopting this embodiment, when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame can be reconstructed according to the image frame stored in the cache. Specifically, the depth information of the key frame can be determined according to the depth information and motion estimation information corresponding to the image pair synchronized in the cache. Through this embodiment, the electronic device can obtain depth information of better quality, which is beneficial to improving the display effect of the blurred captured image generated during the capture.

[0065] like Figure 4 As shown, in one embodiment of the present application, the method for generating a captured image includes the following steps: 401 to 408.

[0066] 401. In blur mode, save the image frame most recently captured by the camera into the cache.

[0067] In some possible implementations, the camera of the electronic device may include: a first camera and a second camera, and the key frame is an image captured by the first camera.

[0068] 402. Get snapshot command.

[0069] The capture instruction may be generated based on the user's operation trigger, or may be generated when a wonderful moment matching a pre-configured action or posture is detected (such as when the user jumps up).

[0070] 403. In response to the snapshot instruction, determine the key frame from the cache.

[0071] In some possible implementations, a picture in the cache is automatically selected as a key frame.

[0072] 404. When the depth information of the key frame cannot be obtained according to the preset method, determine the contour information of the target object according to the key frame.

[0073] The target object may be a person, and the contour information of the target object may be obtained through image processing. In some possible embodiments, 1 may be used to represent the area where the target object is located, and 0 may be used to represent the area outside the target object. For example, the subject segmentation calculation module of the captured key frame may be used to perform related operations: the human body segmentation module based on the AI ​​Segmentation algorithm segments the foreground human body in the key frame through the AI ​​network to obtain the Mask information SegMask of the foreground human body.

[0074] When the depth information of the key frame cannot be obtained according to a preset method, the depth information of the key frame is reconstructed according to the image stored in the buffer.

[0075] In some possible implementations, when an image captured by a second camera synchronized with a key frame is saved in a cache, it is determined that the depth information of the key frame can be obtained; when an image captured by a second camera synchronized with a key frame is not saved in the cache, it is determined that the depth information of the key frame cannot be obtained.

[0076] The cache does not store the image captured by the second camera synchronized with the key frame, which may include the following two situations: the first situation is that the cache does not store the image corresponding to the key frame (such as Figure 5A As shown, the main camera is the first camera, and the auxiliary camera is the second camera. If the fourth frame of the main camera stored in the cache is determined to be a key frame, the auxiliary camera skips the frame sampling and stores it in the cache relative to the main camera, that is, the first frame, the third frame, and the fifth frame of the auxiliary camera are stored in the cache. Since the auxiliary camera only stores the first frame, the third frame, and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. Figure 5A It can be seen that there are images in the cache that are synchronized with the first frame, the third frame, and the fifth frame of the main camera). The second situation is that the image corresponding to the key frame is saved in the cache, but the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is greater than the preset value. For example, if the preset value is 3 milliseconds, when the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 4 milliseconds, it is determined that the depth information of the key frame cannot be obtained according to the preset method. When the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 2 milliseconds, it is determined that the depth information of the key frame can be obtained according to the preset method. For the second situation, see Fig. 6A As shown in , the cache stores the image of the auxiliary camera corresponding to each frame of the image saved by the main camera, and the time difference between the first frame of this camera and the first frame of the auxiliary camera is less than the preset value, then the first frame of the main camera is synchronized with the first frame of the auxiliary camera. The time differences between the second frame, third frame, fourth frame and fifth frame of the main camera and the second frame, third frame, fourth frame and fifth frame of the auxiliary camera are respectively greater than the preset value, then Fig. 6A In the specific implementation, the dual-camera synchronization detection module can be used to detect and confirm the dual-camera synchronization, and according to the detection result of synchronization or non-synchronization, a synchronization Synced or non-synchronization Not Synced flag bit can be output.

[0077] 405. Determine depth information and confidence information corresponding to one or more pairs of images in the synchronized image pairs in the cache.

[0078] There are many methods for binocular depth estimation. You can refer to the implementation of related methods, which will not be described here.

[0079] For example, the depth quality assessment module may be used to perform related operations, such as generating depth quality score information based on the confidence information of the depth calculation module, and the data range may be 0 to 100, wherein a value lower than 60 may be considered as unusable depth information.

[0080] If the key frame position correctly completes the dual-camera frame synchronization and the calculated depth information is available, there is no need to perform the depth information migration operation;

[0081] It should be noted that if the key frame position does not complete the dual-camera frame synchronization, it is necessary to find the 1-2 frames of depth information closest to the key frame position for migration. For example, they can be recorded as CandidateDepth1 and CandidateDepth2. The best case is that there is a frame available before and after the key frame, such as Figure 5B As shown, the intermediate situation is that only the forward or backward has available depth information, such as Figure 6B The worst case is that there is no available depth information frame before or after.

[0082] 406. Determine motion estimation information of the one or more pairs of images relative to the key frame.

[0083] For example, the motion vector calculation module can be used to perform related operations: calculate the motion vector information between frames, specifically, calculate the motion vector information between the image frame and the key frame at the corresponding position of the available depth frame, and output CandidateDepthME1 / CandidateDepthME2 information. If there is only one available depth frame, only one is output; if there is no available depth frame, no output is output.

[0084] 407. Determine the depth information of the key frame according to the contour information, the depth information corresponding to the one or more pairs of images, the confidence information and the motion estimation information.

[0085] For example, the depth information migration calculation module can be used to perform related operations: completing the migration method from adjacent frames to generate the depth information of the current key frame position. Figure 5B and Figure 6B There are two cases. The first case is Figure 5B As shown, the depth information before and after the key frame can be obtained, that is, the forward position of the key frame completes the dual-camera frame synchronization, and the depth information is calculated. The backward position of the key frame also completes the frame synchronization, and the depth information is also calculated. At this time, the forward plus backward migration method can be used. First, CandidateDepthID1 is used in combination with the forward motion estimation information MEFront to perform forward depth information migration to obtain the key frame position depth information KeyFrameDepthFront, and then CandidateDepthID2 information is used in combination with the backward motion estimation information MEBack to perform backward depth information migration to obtain the key frame position depth information KeyFrameDepthBack. By comparing the forward and backward depth information migration results and referring to the key frame human segmentation information SegMask for processing and fusion, the depth information KeyFrameDepth at the key frame position is obtained.

[0086] The core processing steps and methods of the depth information migration module are to use CandidateDepthID1 and MEFront (or CandidateDepthID2 and MEBack) to perform depth information compensation based on motion information. The depth information of the key frame position to be determined here is KeyFrameDepthFront, where KeyFrameDepthFront (i, j) is the depth information to be confirmed at the pixel coordinates i, j. The information is obtained as follows:

[0087] KeyFrameDepthFront(i,j)=CandidateDepthID1((i+MEFront(i,j,1)),(j+MEFront(i,j,2))).

[0088] Among them, MEFront(i,j,1) and MEFront(i,j,2) are the motion vector information in the x and y directions of the pixel coordinates i and j.

[0089] The migration result fusion module can be used to perform the following steps: compare the key frame position depth information KeyFrameDepthFront and the key frame position depth information KeyFrameDepthBack, confirm the difference position of the two depth information, and generate a mask corresponding to the difference position, namely Depth_diff_mask. The data in Depth_diff_mask are different positions of forward estimation and backward estimation. These positions are unreliable depth information; then use the human body segmentation information SegMask and the surrounding neighborhood information for compensation: if the marked bit at the Depth_diff_mask (i, j) position is non-zero, it means that the depth information here is unreliable. The mark of the current position of SegMask (i, j) can be combined, and the position with the same SegMask mark as this position in the surrounding 8 neighborhoods can be found. If it is recorded as m, n position, the depth information of this position is assigned to the current position: Depth_diff_mask (i, j) = Depth_diff_mask (m, n) to complete the compensation and fusion of the unreliable position depth information.

[0090] 408. Generate a blurred snapshot image based on the key frame and the depth information of the key frame.

[0091] When the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame is reconstructed according to the image stored in the buffer.

[0092] In some possible implementations, when an image captured by a second camera synchronized with a key frame is saved in a cache, it is determined that the depth information of the key frame can be obtained; when an image captured by a second camera synchronized with a key frame is not saved in the cache, it is determined that the depth information of the key frame cannot be obtained.

[0093] The cache does not store the image captured by the second camera synchronized with the key frame, which may include the following two situations: the first situation is that the cache does not store the image corresponding to the key frame (such as Figure 5A As shown, the main camera is the first camera, and the auxiliary camera is the second camera. If the fourth frame of the main camera stored in the cache is determined to be a key frame, the auxiliary camera skips the frame sampling and stores it in the cache relative to the main camera, that is, the first frame, the third frame, and the fifth frame of the auxiliary camera are stored in the cache. Since the auxiliary camera only stores the first frame, the third frame, and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. Figure 5AIt can be seen that there are images in the cache that are synchronized with the first frame, the third frame, and the fifth frame of the main camera). The second situation is that the image corresponding to the key frame is saved in the cache, but the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is greater than the preset value. For example, if the preset value is 3 milliseconds, when the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 4 milliseconds, it is determined that the depth information of the key frame cannot be obtained according to the preset method. When the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 2 milliseconds, it is determined that the depth information of the key frame can be obtained according to the preset method. For the second situation, see Fig. 6A As shown in , the cache stores the image of the auxiliary camera corresponding to each frame of the image saved by the main camera, and the time difference between the first frame of this camera and the first frame of the auxiliary camera is less than the preset value, then the first frame of the main camera is synchronized with the first frame of the auxiliary camera. The time differences between the second frame, third frame, fourth frame and fifth frame of the main camera and the second frame, third frame, fourth frame and fifth frame of the auxiliary camera are respectively greater than the preset value, then Fig. 6A In the case of the above, it is determined that the image captured by the second camera synchronized with the key frame is not saved in the cache).

[0094] In some possible implementations, the depth information of the key frame is reconstructed based on one or more pairs of images in the synchronized image pair in the cache. Specifically, the one or more pairs of images may be the preceding images of the key frame; or, the one or more pairs of images may be the succeeding images of the key frame; or, the multiple pairs of images may be the preceding images and succeeding images of the key frame. For example, for Figure 5A In the scene shown, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed based on the nearest forward synchronization frame (the image pair corresponding to the fourth frame) and the subsequent synchronization frame (the image pair corresponding to the fifth frame) of the fourth frame in the cache. Fig. 6A In the scene shown, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed according to the forward synchronization frame of the fourth frame in the cache (the image pair corresponding to the first frame).

[0095] 405. Generate a blurred captured image according to the key frame and the depth information of the key frame.

[0096] By adopting this embodiment, when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame can be determined based on the contour information of the key frame, the depth information corresponding to one or more pairs of synchronized images stored in the cache, the confidence information and the motion estimation information. Through this embodiment, the electronic device can obtain depth information of better quality, which is conducive to improving the display effect of the blurred captured image generated during the capture.

[0097] As an example, see Figure 7, is a schematic diagram of the structure of an electronic device 700 provided in an embodiment of the present application. Figure 7 As shown, the electronic device 700 may include: a processor 701, a communication module 702, a display screen 703, etc.

[0098] The processor 701 may include one or more processing units, and different processing units may be independent devices or integrated into one or more processors 701. The controller may be the nerve center and command center of the electronic device 700. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions. A memory may also be provided in the processor 701 for storing instructions and data.

[0099] In some embodiments, the processor 701 may include one or more interfaces. The interface may include an integrated circuit interface, an integrated circuit built-in audio interface, a pulse code modulation interface, a universal asynchronous receiver and transmitter interface, a mobile industry processor interface, a general purpose input and output interface, a subscriber identity module interface, and / or a universal serial bus interface 711, etc.

[0100] The electronic device 700 implements the display function through a GPU, a display screen 703, and an application processor 701. The GPU is a microprocessor for image processing, which connects the display screen 703 and the application processor 701. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 701 may include one or more GPUs, which execute program instructions to generate or change display information. The display screen 703 is used to display images, video streams, etc. The communication module 702 may include an antenna x, an antenna y, a mobile communication module 702A, and / or a wireless communication module 702B. Take the communication module 702 as an example, which includes an antenna x, an antenna y, a mobile communication module 702A, and a wireless communication module 702B at the same time. In addition, the receiving module provided in the embodiment of the present application may also be arranged in the communication module 702.

[0101] The wireless communication function of the electronic device 700 can be implemented through antenna x, antenna y, mobile communication module 702A, wireless communication module 702B, a modem processor, and a baseband processor.

[0102] The mobile communication module 702A can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 700. The mobile communication module 702A may include at least one filter, a switch, a power amplifier, a low noise amplifier group (LNA), etc. The mobile communication module 702A can receive electromagnetic waves from the antenna x, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 702A can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna x. In some embodiments, at least some of the functional modules of the mobile communication module 702A can be set in the processor 701. In some embodiments, at least some of the functional modules of the mobile communication module 702A can be set in the same device as at least some of the modules of the processor 701.

[0103] The application processor outputs sound signals through an audio device (not limited to a speaker 706A, a receiver 706B, etc.), or displays an image or video stream through a display screen 703. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 701 and may be provided in the same device as the mobile communication module 702A or other functional modules.

[0104] The wireless communication module 702B can provide a wireless communication solution applied to the electronic device 700. The wireless communication module 702B can be one or more devices integrating at least one communication processing module. The wireless communication module 702B receives electromagnetic waves via the antenna y, modulates the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 701. The wireless communication module 702B can also receive a signal to be sent from the processor 701, modulate the frequency, amplify it, and convert it into an electromagnetic wave for radiation through the antenna y.

[0105] The antenna x of the electronic device 700 is coupled to the mobile communication module 702A, and the antenna y is coupled to the wireless communication module 702B, so that the electronic device 700 can communicate with the network and other devices through wireless communication technology.

[0106] The electronic device 700 may further include an external memory interface 710, an internal memory 704, a universal serial bus (USB) interface 711, a charging management module 712, a power management module 713, a battery 714, an audio module 706, a speaker 706A, a receiver 706B, a microphone 706C, an earphone interface 706D, a sensor module 705, a button 709, a motor, an indicator 708, a camera 707, and a subscriber identification module (SIM) card interface. The electronic device 700 may implement audio functions through the audio module 706, the speaker 706A, the receiver 706B, the microphone 706C, the earphone interface 706D, and the application processor 701. For example, music playing, recording, etc. The sensor module 705 in the electronic device 700 may implement sensing and / or acquisition functions for different signals.

[0107] The electronic device provided by the embodiment of the present application is introduced above. It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 700. In other embodiments, the electronic device 700 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0108] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0109] The hardware system of the electronic device 700 is described in detail above, and the software system of the electronic device 700 is introduced below.

[0110] Figure 8 It is a schematic diagram of a software system of an electronic device provided in an embodiment of the present application.

[0111] like Figure 8 As shown, the system architecture may include an application layer 810 , an application framework layer 820 , a hardware abstraction layer 830 , a driver layer 840 and a hardware layer 850 .

[0112] The application layer 810 may include a camera application.

[0113] Optionally, the application layer 810 may also include camera applications, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0114] The application framework layer 820 provides an application programming interface (API) and a programming framework for the applications in the application layer; the application framework layer may include some predefined functions.

[0115] For example, the application framework layer 820 may include a camera access interface, which may include camera management and camera equipment. The camera management may be used to provide an access interface for managing the camera, and the camera equipment may be used to provide an interface for accessing the camera.

[0116] The hardware abstraction layer 830 is used to abstract the hardware. For example, the hardware abstraction layer may include a camera abstraction layer and other hardware device abstraction layers; the camera abstraction layer may include camera device 1, camera device 2, etc.; the camera hardware abstraction layer may be connected to the camera algorithm library, and the camera hardware abstraction layer may call the algorithm in the camera algorithm library.

[0117] The driver layer 840 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera device driver.

[0118] The hardware layer 850 may include an image sensor, an image signal processor (ISP) and other hardware devices; wherein the sensor can be used to collect Raw images; the Raw images collected by the sensor can be sent to the image signal processor for image processing and output the processed images; and the processed images are transmitted to the camera application of the application layer; wherein the image signal processor may include an automatic exposure module, and the automatic exposure module is used to perform automatic exposure processing; the automatic exposure module includes a control module, and the control module is used to execute the relevant algorithms of the method for generating captured images provided in the embodiment of the present application: in blur mode, the image frame most recently collected by the camera is saved in a cache; a capture instruction is obtained; in response to the capture instruction, a key frame is determined from the cache; when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame is reconstructed according to the image saved in the cache; a blurred capture image is generated according to the key frame and the depth information of the key frame.

[0119] The present application also provides a computer program product, which, when executed by a processor, implements the interactive method of any method embodiment in the present application.

[0120] The present application also provides a computer-readable storage medium, which includes instructions. When the instructions are executed on an electronic device, the electronic device executes an interactive method as in any of the above method embodiments. The specific steps of the interactive method refer to the description in the above method embodiments, which will not be repeated here.

[0121] It should be noted that although the embodiments of the present application are described using the Android system as an example, its basic principles are also applicable to electronic devices based on operating systems such as iOS and Windows.

[0122] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic device, storage medium and program product described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] In the several embodiments provided in the present application, it should be understood that the disclosed methods, electronic devices, storage media and program products can be implemented in other ways. For example, the embodiments of the electronic devices described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0127] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0128] In addition, the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0129] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM) and other media that can store program codes.

[0130] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims. In short, the above are only preferred embodiments of the technical solution of the present application, and are not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating a captured image, characterized in that: Applied to electronic equipment, the method comprises: In blur mode, the image frame most recently captured by the camera is saved in the cache; Get the capture command; In response to the snapshot instruction, determining a key frame from the buffer; When the depth information of the key frame cannot be obtained according to the preset method, reconstructing the depth information of the key frame according to the image stored in the buffer; A blurred captured image is generated according to the key frame and the depth information of the key frame.

2. The method according to claim 1, characterized in that The camera of the electronic device includes: a first camera and a second camera, and the key frame is an image captured by the first camera.

3. The method according to claim 2, characterized in that The preset method includes: When the image captured by the second camera synchronized with the key frame is saved in the cache, it is determined that the depth information of the key frame can be obtained; when the image captured by the second camera synchronized with the key frame is not saved in the cache, it is determined that the depth information of the key frame cannot be obtained.

4. The method according to claim 3, characterized in that The reconstructing the depth information of the key frame according to the image stored in the cache includes: The depth information of the key frame is reconstructed according to one or more pairs of images in the synchronized image pairs in the cache.

5. The method according to claim 4, characterized in that The reconstructing the depth information of the key frame according to one or more pairs of images in the synchronized image pairs in the cache includes: Determining depth information corresponding to the one or more pairs of images respectively; Determining motion estimation information of the one or more pairs of images relative to the key frame; The depth information of the key frame is determined according to the depth information and the motion estimation information respectively corresponding to the one or more pairs of images.

6. The method according to claim 4, characterized in that The reconstructing the depth information of the key frame according to one or more pairs of images in the synchronized image pairs in the cache includes: Determine contour information of the target object according to the key frame; Determining confidence information corresponding to the one or more pairs of images respectively; Determining depth information corresponding to the one or more pairs of images respectively; Determining motion estimation information of the one or more pairs of images relative to the key frame; The depth information of the key frame is determined according to the contour information, and the confidence information, the depth information and the motion estimation information respectively corresponding to the one or more pairs of image frames.

7. The method according to any one of claims 4 to 6, characterized in that: The one or more pairs of images are preceding images of the key frame; or, The one or more pairs of images are subsequent images of the key frame; or, The multiple pairs of images are preceding images and succeeding images of the key frame.

8. An electronic device, characterized in that: The device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises executable instructions, and when the executable instructions are executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 7.

Images