Method for generating snapshot image, electronic device, storage medium and program product

Abstract

Embodiments of the present application provide a method for generating a snapshot image, an electronic device, a storage medium and a program product, which are applied to the electronic device. The method comprises: saving an image frame recently collected by a camera into a cache in a blurring mode; obtaining a snapshot instruction; determining a key frame from the cache in response to the snapshot instruction; when depth information of the key frame cannot be obtained according to a preset method, reconstructing the depth information of the key frame according to the image saved in the cache; and generating a blurred snapshot image according to the key frame and the depth information of the key frame. According to the embodiment, when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame can be reconstructed according to the image frame saved in the cache. The electronic device can obtain good quality depth information, which is conducive to improving the display effect of the blurred snapshot image generated when the snapshot is taken.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a method for generating a snapshot image, an electronic device, a storage medium and a program product. BACKGROUND

[0002] With the rapid development of electronic technology, electronic devices such as mobile phones are applied more and more widely, and users have higher and higher requirements for images obtained by using electronic devices. For example, when a user takes a snapshot of some wonderful moment, a portrait or a close-up image, the user hopes that the snapshot image can present a blur effect. In the prior art, the blur effect of the snapshot image is not ideal.

[0003] Therefore, how to improve the blur effect of the snapshot image becomes a technical problem to be solved. SUMMARY

[0004] The present application provides a method for generating a snapshot image, an electronic device, a storage medium and a program product, which can improve the blur effect of the snapshot image.

[0005] In a first aspect, the present application provides a method for generating a snapshot image, applied to an electronic device, the method comprising: saving an image frame most recently collected by a camera into a cache in a blur mode; obtaining a snapshot instruction; determining a key frame from the cache in response to the snapshot instruction; when depth information of the key frame cannot be obtained according to a preset method, reconstructing the depth information of the key frame according to the image saved in the cache; and generating a blurred snapshot image according to the key frame and the depth information of the key frame.

[0006] In this embodiment, when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame can be reconstructed according to the image frame saved in the cache, and the electronic device can obtain good quality depth information, which is conducive to improving the display effect of the blurred snapshot image generated when taking a snapshot.

[0007] In some possible embodiments, the key frame can be the clearest image frame in the cache.

[0008] In some possible embodiments, obtaining the snapshot instruction can be: obtaining a snapshot instruction triggered by a user; or, obtaining the snapshot instruction can be: obtaining a snapshot instruction automatically triggered by the electronic device when an image matching a preset action or posture is detected.

[0009] In some possible embodiments, the camera of the electronic device comprises: a first camera and a second camera, and the key frame is an image collected by the first camera.

[0010] In some embodiments of the first aspect, the preset method comprises: determining that the depth information of the key frame can be obtained when the image captured by the second camera synchronized with the key frame is saved in the cache; and determining that the depth information of the key frame cannot be obtained when the image captured by the second camera synchronized with the key frame is not saved in the cache.

[0011] In this embodiment, the preset method is limited. When the depth information of the key frame can be obtained according to the preset method, the depth information of the key frame is determined according to the image captured by the second camera synchronized with the key frame saved in the cache. When the image captured by the second camera synchronized with the key frame is not saved in the cache, it is determined that the depth information of the key frame cannot be obtained according to the preset method. In this case, the depth information of the key frame is reconstructed according to the image frames saved in the cache.

[0012] In some embodiments of the first aspect, the depth information of the key frame is reconstructed according to the image saved in the cache, comprising: reconstructing the depth information of the key frame according to one or more pairs of image synchronized in the cache.

[0013] In some embodiments of the first aspect, the depth information of the key frame is reconstructed according to one or more pairs of image synchronized in the cache, comprising: determining the depth information corresponding to the one or more pairs of image respectively; determining the motion estimation information of the one or more pairs of image relative to the key frame; and determining the depth information of the key frame according to the depth information corresponding to the one or more pairs of image respectively and the motion estimation information.

[0014] In this embodiment, how to reconstruct the depth information of the key frame according to one or more pairs of image synchronized in the cache is limited. Specifically, the depth information of the key frame is determined according to the depth information corresponding to the one or more pairs of image synchronized in the cache and the motion estimation information.

[0015] In some embodiments of the first aspect, the depth information of the key frame is reconstructed according to the image saved in the cache, comprising: determining the contour information of the target object according to the key frame; determining the confidence information corresponding to the one or more pairs of image respectively; determining the depth information corresponding to the one or more pairs of image respectively; determining the motion estimation information of the one or more pairs of image relative to the key frame; and determining the depth information of the key frame according to the contour information, the confidence information, the depth information and the motion estimation information corresponding to the one or more pairs of image respectively.

[0016] The embodiment defines how to reconstruct the depth information of the key frame by using one or more pairs of images in the pair of synchronized images in the cache. Specifically, the depth information of the key frame is determined according to the confidence information, the depth information and the motion estimation information corresponding to the pair of synchronized images in the cache, and according to the contour information of the target object determined by the key frame.

[0017] In combination with the first aspect, in some implementations of the first aspect, the one or more pairs of images are the preceding images of the key frame; or the one or more pairs of images are the following images of the key frame; or the multiple pairs of images are the preceding images and the following images of the key frame.

[0018] In the second aspect, the embodiments of the present application provide an electronic device, including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the steps of any one of the first aspect.

[0019] In the third aspect, the embodiments of the present application provide a computer readable storage medium, including a stored program, wherein when the program runs, the device where the computer readable storage medium is located is controlled to execute the method of any one of the first aspect.

[0020] In the fourth aspect, the embodiments of the present application provide a computer program product, including executable instructions, when the executable instructions are executed on a computer, the computer is caused to execute the method of any one of the first aspect.

[0021] It should be understood that the technical solutions provided by the above-mentioned second aspect to fourth aspect have technical features corresponding to the method for generating a snapshot image provided in the first aspect and its possible implementations, and therefore can achieve similar beneficial effects, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a schematic diagram of a blurred snapshot image generated by an embodiment of the present application;

[0023] Figure 2 is a flowchart of a method for generating a snapshot image provided by an embodiment of the present application;

[0024] Figure 3 is a flowchart of a method for generating a snapshot image provided by an embodiment of the present application;

[0025] Figure 4 is a flowchart of a method for generating a snapshot image provided by an embodiment of the present application;

[0026] Figure 5A is data saved in a cache of an embodiment of the present application, and a key frame sketch;

[0027] Figure 5B is data saved in a cache of an embodiment of the present application, and a key frame corresponding depth information generation sketch;

[0028] Figure 6A is data saved in a cache of an embodiment of the present application, and a key frame sketch;

[0029] Figure 6B is data saved in a cache of an embodiment of the present application, and a key frame corresponding depth information generation sketch;

[0030] Figure 7 is a structure sketch of a terminal provided by an embodiment of the present application;

[0031] Figure 8 is a software structure block diagram of a terminal provided by an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to better understand the technical solutions of the present application, the embodiments of the present application are described in detail below in combination with the drawings.

[0033] In order to facilitate clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", etc. For example, the first instruction and the second instruction are used to distinguish different user instructions, and do not limit the order. Those skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not necessarily mean different.

[0034] It should be noted that in the present application, "exemplarily" or "for example" and the like are used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplarily" or "for example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplarily" or "for example" and the like is intended to present the relevant concept in a specific manner.

[0035] The generation method of the snapshot image provided in the embodiments of the present application can be applied to a terminal with a snapshot function, such as a mobile phone, a tablet computer, a personal computer (PC), a personal digital assistant (PDA), a smart watch, a netbook, a wearable electronic device, an augmented reality (AR) device, a virtual reality (VR) device, and the like. It should be noted that the present application does not limit the specific type of the electronic device. For simplicity, the electronic device is taken as a mobile phone in the embodiments of the present application.

[0036] The idea of the present application includes: in the blur mode, saving the image frame most recently collected by the camera into a cache; triggering the generation of a snapshot instruction when a user's triggering snapshot instruction is acquired or when a situation such as a wonderful moment matching a preconfigured action or posture is detected (for example, when the user jumps up); in response to the snapshot instruction, determining a key frame from the cache; when the depth information of the key frame cannot be obtained according to a preset method, reconstructing the depth information of the key frame according to the image saved in the cache; and generating a blurred snapshot image according to the key frame and the depth information of the key frame, such as Figure 1 is a blurred snapshot image generated after the user jumps up by using the generation method of the snapshot image provided in the embodiments of the present application.

[0037] As shown in Figure 2 In an embodiment of the present application, the generation method of the snapshot image includes the following steps: 201 to 205.

[0038] 201. In the blur mode, saving the image frame most recently collected by the camera into a cache.

[0039] The cache can be a circular queue, and multiple image frames can be cached in the circular queue. In some possible embodiments, the camera of the electronic device can include a first camera and a second camera, and the key frame is an image collected by the first camera.

[0040] 202. Acquiring a snapshot instruction.

[0041] The snapshot instruction can be triggered to be generated according to the operation of the user, or the snapshot instruction can be triggered to be generated when a situation such as a wonderful moment matching a preconfigured action or posture is detected (for example, when the user jumps up).

[0042] 203. In response to the snapshot instruction, determining a key frame from the cache.

[0043] In some possible embodiments, one picture in the cache is automatically selected as the key frame.

[0044] 204. When the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame is reconstructed according to the image stored in the cache.

[0045] In some possible implementation manners, when the image captured by the second camera synchronized with the key frame is stored in the cache, it is determined that the depth information of the key frame can be obtained; when the image captured by the second camera synchronized with the key frame is not stored in the cache, it is determined that the depth information of the key frame cannot be obtained.

[0046] The image captured by the second camera synchronized with the key frame not being stored in the cache can include the following two cases: the first case is that the image corresponding to the key frame is not stored in the cache (for example, as shown in FIG. 1, the fourth frame of the main camera is the key frame, and the image corresponding to the fourth frame is not stored in the cache). Figure 5A As shown in FIG. 1, the main camera is the first camera, and the auxiliary camera is the second camera. If it is determined that the fourth frame of the main camera stored in the cache is the key frame. The auxiliary camera is sampled by frame skipping relative to the main camera and is stored in the cache, that is, the first frame, the third frame and the fifth frame of the auxiliary camera are stored in the cache. Since the auxiliary camera only stores the first frame, the third frame and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. As can be seen from FIG. 1, there is an image synchronized with the first frame, the third frame and the fifth frame of the main camera in the cache. Figure 5A As can be seen from FIG. 1, there is an image synchronized with the first frame, the third frame and the fifth frame of the main camera in the cache. The second case is that the image corresponding to the key frame is stored in the cache, but the shooting time difference between the image corresponding to the key frame stored in the cache and the key frame is greater than a preset value. For example, if the preset value is 3 milliseconds, when the shooting time difference between the image corresponding to the key frame stored in the cache and the key frame is 4 milliseconds, it is determined that the depth information of the key frame cannot be obtained according to the preset method. When the shooting time difference between the image corresponding to the key frame stored in the cache and the key frame is 2 milliseconds, it is determined that the depth information of the key frame can be obtained according to the preset method. The second case is shown in FIG. 2. Figure 6A As shown in FIG. 1, the main camera is the first camera, and the auxiliary camera is the second camera. If it is determined that the fourth frame of the main camera stored in the cache is the key frame. The auxiliary camera is sampled by frame skipping relative to the main camera and is stored in the cache, that is, the first frame, the third frame and the fifth frame of the auxiliary camera are stored in the cache. Since the auxiliary camera only stores the first frame, the third frame and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. As can be seen from FIG. 1, there is an image synchronized with the first frame, the third frame and the fifth frame of the main camera in the cache. Figure 6A As can be seen from FIG. 1, there is an image synchronized with the first frame, the third frame and the fifth frame of the main camera in the cache. The second case is that the image corresponding to the key frame is stored in the cache, but the shooting time difference between the image corresponding to the key frame stored in the cache and the key frame is greater than a preset value. For example, if the preset value is 3 milliseconds, when the shooting time difference between the image corresponding to the key frame stored in the cache and the key frame is 4 milliseconds, it is determined that the depth information of the key frame cannot be obtained according to the preset method. When the shooting time difference between the image corresponding to the key frame stored in the cache and the key frame is 2 milliseconds, it is determined that the depth information of the key frame can be obtained according to the preset method. The second case is shown in FIG. 2.

[0047] In some possible implementation manners, the depth information of the key frame is reconstructed according to one or more pairs of images in the synchronized image pairs in the cache. Specifically, the one or more pairs of images are the preceding image of the key frame, or the one or more pairs of images are the following image of the key frame, or the one or more pairs of images are the preceding image and the following image of the key frame. For example, for the scene shown in FIG. 4, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed according to the preceding synchronized frame (the image pair corresponding to the fourth frame) and the following synchronized frame (the image pair corresponding to the fifth frame) of the fourth frame in the cache. Figure 5A For the scene shown in FIG. 4, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed according to the preceding synchronized frame (the image pair corresponding to the fourth frame) and the following synchronized frame (the image pair corresponding to the fifth frame) of the fourth frame in the cache. Figure 6A For the scene shown in FIG. 4, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed according to the preceding synchronized frame (the image pair corresponding to the fourth frame) and the following synchronized frame (the image pair corresponding to the fifth frame) of the fourth frame in the cache.

[0048] 205. generating a blurred snapshot image according to the key frame and the depth information of the key frame.

[0049] With this embodiment, when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame can be reconstructed according to the image frames saved in the cache, so that the electronic device can obtain depth information of better quality, and the display effect of the blurred snapshot image generated when the snapshot is taken can be improved.

[0050] As shown in FIG. 3, in an embodiment of the present application, the method for generating a snapshot image includes the following steps: 301 to 307. Figure 3

[0051] 301. In the blurred mode, saving the image frame most recently collected by the camera into the cache.

[0052] In some possible implementation manners, the camera of the electronic device can include a first camera and a second camera, and the key frame is an image collected by the first camera.

[0053] 302. obtaining a snapshot instruction.

[0054] The snapshot instruction can be triggered to be generated according to the operation of a user, or the snapshot instruction can be triggered to be generated when a situation such as a wonderful moment matching a preconfigured action or posture (for example, when the user jumps up) occurs.

[0055] 303. determining a key frame from the cache in response to the snapshot instruction.

[0056] In some possible implementation manners, one image in the cache is automatically selected as the key frame.

[0057] 304. when the depth information of the key frame cannot be obtained according to the preset method, reconstructing the depth information of the key frame according to the image saved in the cache.​

[0058] In some possible implementation, when the image captured by the second camera synchronized with the key frame is saved in the cache, it is determined that the depth information of the key frame can be obtained; when the image captured by the second camera synchronized with the key frame is not saved in the cache, it is determined that the depth information of the key frame cannot be obtained.

[0059] The case that the image captured by the second camera synchronized with the key frame is not saved in the cache can include two cases: the first case is that the image corresponding to the key frame is not saved in the cache (for example, as shown in FIG. 4, the image corresponding to the fourth frame of the main camera is not saved in the cache); the second case is that the image corresponding to the key frame is saved in the cache, but the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is greater than a preset value (for example, as shown in FIG. 5, the shooting time difference between the image corresponding to the fourth frame of the main camera saved in the cache and the fourth frame of the main camera is 4 ms, which is greater than the preset value 3 ms). Figure 5A As shown in FIG. 4, the main camera is the first camera, and the auxiliary camera is the second camera. It is determined that the fourth frame saved in the cache by the main camera is the key frame. The auxiliary camera is sampled and saved in the cache by frame skipping relative to the main camera, that is, the first frame, the third frame and the fifth frame of the auxiliary camera are saved in the cache. Since the auxiliary camera only saves the first frame, the third frame and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. It is determined that the depth information of the key frame cannot be obtained according to the preset method, because there is no image corresponding to the key frame in the cache. Figure 5A It can be known that there is an image synchronized with the first frame, the third frame and the fifth frame of the main camera in the cache. The second case is that the image corresponding to the key frame is saved in the cache, but the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is greater than a preset value. For example, if the preset value is 3 ms, when the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 4 ms, it is determined that the depth information of the key frame cannot be obtained according to the preset method. When the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 2 ms, it is determined that the depth information of the key frame can be obtained according to the preset method. The second case is shown in FIG. 5. Figure 6A As shown in FIG. 5, the image of the auxiliary camera corresponding to each frame of the image saved by the main camera is stored in the cache. The time difference between the first frame of the main camera and the first frame of the auxiliary camera is less than the preset value, so the first frame of the main camera is synchronized with the first frame of the auxiliary camera. The time difference between the second frame, the third frame, the fourth frame and the fifth frame of the main camera and the second frame, the third frame, the fourth frame and the fifth frame of the auxiliary camera is greater than the preset value, so the second frame, the third frame, the fourth frame and the fifth frame of the main camera are not synchronized with the second frame, the third frame, the fourth frame and the fifth frame of the auxiliary camera. Figure 6A The case shown in FIG. 5 determines that the image captured by the second camera synchronized with the key frame is not saved in the cache.

[0060] 305. Determine the motion estimation information of the one or more pairs of images relative to the key frame.

[0061] In some possible implementation, the depth information of the key frame is reconstructed according to one or more pairs of images in the pair of synchronized images in the cache. Specifically, the one or more pairs of images can be the previous images of the key frame; or the one or more pairs of images are the subsequent images of the key frame; or the one or more pairs of images are the previous images and the subsequent images of the key frame. For example, for the case shown in FIG. 4, the depth information of the key frame is reconstructed according to the first frame and the third frame of the auxiliary camera saved in the cache.Figure 5A As shown in the scenario, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed according to the forward synchronized frame (the image pair corresponding to the first frame) and the backward synchronized frame (the image pair corresponding to the fifth frame) of the fourth frame in the cache. Figure 6A As shown in the scenario, since the fourth frame is a key frame, the depth information of the key frame can be reconstructed according to the forward synchronized frame (the image pair corresponding to the first frame) and the backward synchronized frame (the image pair corresponding to the fifth frame) of the fourth frame in the cache.

[0062] 306. Determine the depth information of the key frame according to the depth information and the motion estimation information corresponding to one or more image pairs.

[0063] 307. Generate a blurred snapshot image according to the key frame and the depth information of the key frame.

[0064] With this embodiment, when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame can be reconstructed according to the image frames saved in the cache, specifically, the depth information of the key frame can be determined according to the depth information and the motion estimation information corresponding to the synchronized image pairs in the cache, and through this embodiment, the electronic device can obtain depth information of good quality, which is conducive to improving the display effect of the blurred snapshot image generated when the snapshot is taken.

[0065] As shown in the scenario, in an embodiment of the present application, the method for generating a snapshot image includes the following steps: 401 to 408. Figure 4

[0066] 401. In the blur mode, save the image frame recently captured by the camera into the cache.

[0067] In some possible embodiments, the camera of the electronic device can include a first camera and a second camera, and the key frame is an image captured by the first camera.

[0068] 402. Obtain a snapshot instruction.

[0069] The snapshot instruction can be triggered to be generated according to the operation of the user, or can be triggered to be generated when a wonderful moment matching a preconfigured action or posture is detected (for example, when the user jumps up).

[0070] 403. In response to the snapshot instruction, determine a key frame from the cache.

[0071] In some possible embodiments, one picture in the cache is automatically selected as the key frame.

[0072] 404. When the depth information of the key frame cannot be obtained according to the preset method, determine the contour information of the target object according to the key frame.

[0073] ​The target object can be a person, and the contour information of the target object can be obtained through image processing. In some possible embodiments, the area where the target object is located can be represented by 1, and the area where the target object is not located can be represented by 0. For example, the subject segmentation calculation module of the snapshot key frame can be used for related operations: a human body segmentation module based on an AI Segmentation algorithm, the foreground human body in the key frame is segmented through an AI network, and Mask information SegMask of the foreground human body is obtained.

[0074] When the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame is reconstructed according to the image saved in the cache.

[0075] In some possible embodiments, when the image captured by the second camera synchronized with the key frame is saved in the cache, it is determined that the depth information of the key frame can be obtained; and when the image captured by the second camera synchronized with the key frame is not saved in the cache, it is determined that the depth information of the key frame cannot be obtained.

[0076] When the image captured by the second camera synchronized with the key frame is not saved in the cache, it can include the following two cases: the first case is that the image corresponding to the key frame is not saved in the cache (for example, the image corresponding to the fourth frame saved in the cache is not the key frame, as shown in FIG. 6A, the image corresponding to the fourth frame saved in the cache is not the key frame, as shown in FIG. 6B, and the image corresponding to the fourth frame saved in the cache is not the key frame, as shown in FIG. 6C). Figure 5A As shown in FIG. 6A, the main camera is the first camera, and the auxiliary camera is the second camera. If it is determined that the fourth frame saved in the cache by the main camera is the key frame. The auxiliary camera is sampled by frame skipping and saved in the cache relative to the main camera, that is, the first frame, the third frame and the fifth frame of the auxiliary camera are saved in the cache. Since the auxiliary camera only saves the first frame, the third frame and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. As shown in FIG. 6B, the first frame, the third frame and the fifth frame of the main camera are saved in the cache, and the image corresponding to the fourth frame of the main camera is saved in the cache. Since the auxiliary camera only saves the first frame, the third frame and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. As shown in FIG. 6C, the first frame, the third frame and the fifth frame of the main camera are saved in the cache, and the image corresponding to the fourth frame of the main camera is saved in the cache. Since the auxiliary camera only saves the first frame, the third frame and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. Figure 5A It can be seen that there is an image synchronized with the first frame, the third frame and the fifth frame of the main camera in the cache. The second case is that the image corresponding to the key frame is saved in the cache, but the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is greater than a preset value. For example, if the preset value is 3 milliseconds, when the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 4 milliseconds, it is determined that the depth information of the key frame cannot be obtained according to the preset method. When the shooting time difference between the image corresponding to the key frame saved in the cache and the key frame is 2 milliseconds, it is determined that the depth information of the key frame can be obtained according to the preset method. The second case is shown in FIG. 6D. Figure 6A As shown in FIG. 6D, the image of the auxiliary camera corresponding to each image saved by the main camera is stored in the cache. Among them, the time difference between the first frame of the main camera and the first frame of the auxiliary camera is less than the preset value, so the first frame of the main camera is synchronized with the first frame of the auxiliary camera. The time difference between the second frame, the third frame, the fourth frame and the fifth frame of the main camera and the second frame, the third frame, the fourth frame and the fifth frame of the auxiliary camera is greater than the preset value, respectively.Figure 6A In the case of the determination that the second camera image captured in synchronization with the key frame is not saved in the cache, the dual-camera synchronization detection module is used to detect and confirm the dual-camera synchronization, and according to the detection result, the synchronized or non-synchronized flag bit is output.

[0077] 405. Determine the depth information and confidence information corresponding to one or more images in the pair of synchronized images in the cache.

[0078] There are many methods for binocular depth estimation, and relevant methods can be referred to for implementation, which will not be described here.

[0079] For example, the depth quality evaluation module can be used to perform relevant operations, such as generating depth quality score information according to the confidence information of the depth calculation module, and the data range can be 0-100, wherein the value less than 60 can be considered as unusable depth information.

[0080] If the dual-camera frame synchronization is completed correctly at the key frame position and the calculated depth information is available, the depth information migration operation is not required.

[0081] It should be noted that if the dual-camera frame synchronization is not completed at the key frame position, the nearest 1-2 frames of depth information to the key frame position need to be found for migration, such as CandidateDepth1 and CandidateDepth2; the best case is that there is a frame available before and after the key frame, as shown in FIG. 6, the intermediate case is that only the forward or backward has available depth information available, as shown in FIG. 7, and the worst case is that there is no available depth information frame before and after. Figure 5B Figure 6B

[0082] 406. Determine the motion estimation information of the one or more images relative to the key frame.

[0083] For example, the motion vector calculation module can be used to perform relevant operations: calculate the motion vector information between frames, specifically, calculate the motion vector information between the image frame at the position corresponding to the available depth frame and the key frame, and output the CandidateDepthME1 / CandidateDepthME2 information, if there is only one available depth frame, only one is output, and if there is no available depth frame, no output.

[0084] 407. Determine the depth information of the key frame according to the contour information, the depth information, the confidence information, and the motion estimation information corresponding to the one or more images.

[0085] ​​For example, the depth information migration calculation module can be used to perform the following operations: complete the migration from the adjacent frame to generate the depth information of the current key frame position. Figure 5B and Figure 6B For the two cases, the first case is shown in Figure 5B , the depth information before and after the key frame can be obtained, that is, the depth information of the forward position of the key frame is calculated, the frame synchronization is completed, the depth information of the backward position of the key frame is also calculated, and the frame synchronization is completed. At this time, the forward and backward migration method can be used. First, CandidateDepthID1 is used in combination with forward motion estimation information MEFront to perform forward depth information migration to obtain the depth information KeyFrameDepthFront of the key frame position. Then, CandidateDepthID2 information is used in combination with backward motion estimation information MEBack to perform backward depth information migration to obtain the depth information KeyFrameDepthBack of the key frame position. By comparing the forward and backward depth information migration results and referring to the human body segmentation information SegMask of the key frame, the depth information KeyFrameDepth of the key frame position is obtained through processing and fusion.

[0086] The core processing steps and methods of the depth information migration module are to use CandidateDepthID1 and MEFront (or CandidateDepthID2 and MEBack) to perform depth information compensation based on motion information. The key frame position depth information to be determined is KeyFrameDepthFront, wherein KeyFrameDepthFront(i,j) is the depth information to be confirmed at the pixel coordinates i and j. The information is obtained in the following manner:

[0087] KeyFrameDepthFront(i,j) = CandidateDepthID1((i+MEFront(i,j,1)),(j+MEFront(i,j,2))).

[0088] Wherein MEFront(i,j,1) and MEFront(i,j,2) are the motion vector information of the pixel coordinates i and j in the x direction and the y direction.

[0089] The following steps can be performed by the migration result fusion module: comparing the key frame position depth information KeyFrameDepthFront and the key frame position depth information KeyFrameDepthBack, confirming the difference positions of the two depth information, and generating a mask corresponding to the difference positions, i.e., Depth_diff_mask, the data in Depth_diff_mask is the position where the forward estimation and the backward estimation are different, and these positions are unreliable depth information; then, the human body segmentation information SegMask and the surrounding neighborhood information are used for compensation: if the Depth_diff_mask(i,j) position is marked as non-zero, it means that the depth information at this position is unreliable, the current position mark of SegMask(i,j) can be combined, and the positions with the same SegMask mark in the surrounding 8-neighborhood are searched, and the depth information of the position is assigned to the current position: Depth_diff_mask(i,j)=Depth_diff_mask(m,n), to complete the compensation and fusion of the unreliable position depth information.

[0090] 408. generating a blurred snapshot image according to the key frame and the depth information of the key frame.

[0091] When the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame is reconstructed according to the image saved in the cache.

[0092] In some possible implementations, when the image captured by the second camera synchronized with the key frame is saved in the cache, it is determined that the depth information of the key frame can be obtained; when the image captured by the second camera synchronized with the key frame is not saved in the cache, it is determined that the depth information of the key frame cannot be obtained.

[0093] When the image captured by the second camera synchronized with the key frame is not saved in the cache, it can include the following two cases: the first case is that the image corresponding to the key frame is not saved in the cache (for example, the fourth frame of the main camera is determined as the key frame, and the first frame, the third frame and the fifth frame of the auxiliary camera are saved in the cache, and the image corresponding to the fourth frame of the main camera is not saved in the cache); the second case is that the image corresponding to the key frame is saved in the cache (for example, the fourth frame of the main camera is determined as the key frame, and the first frame, the third frame and the fifth frame of the auxiliary camera are saved in the cache, and the image corresponding to the fourth frame of the main camera is saved in the cache). Figure 5A As shown in the figure, the main camera is the first camera, and the auxiliary camera is the second camera. If it is determined that the fourth frame of the main camera saved in the cache is the key frame. The auxiliary camera is frame-sampled and saved in the cache relative to the main camera, that is, the first frame, the third frame and the fifth frame of the auxiliary camera are saved in the cache. Since the auxiliary camera only saves the first frame, the third frame and the fifth frame in the cache, there is no image frame corresponding to the fourth frame of the main camera in the cache. Figure 5AIt can be seen that the cache stores images synchronized with the first frame, the third frame and the fifth frame of the main camera. The second case is that the cache stores images corresponding to the key frame, but the shooting time difference between the images corresponding to the key frame stored in the cache and the key frame is greater than a preset value. For example, if the preset value is 3 milliseconds, when the shooting time difference between the images corresponding to the key frame stored in the cache and the key frame is 4 milliseconds, it is determined that the depth information of the key frame cannot be obtained according to the preset method. When the shooting time difference between the images corresponding to the key frame stored in the cache and the key frame is 2 milliseconds, it is determined that the depth information of the key frame can be obtained according to the preset method. The second case is shown in Figure 6A As shown in FIG. 7, the cache stores images of the auxiliary camera corresponding to each frame of image stored by the main camera, wherein the time difference between the first frame of the main camera and the first frame of the auxiliary camera is less than a preset value, and the first frame of the main camera is synchronized with the first frame of the auxiliary camera. The time difference between the second frame, the third frame, the fourth frame and the fifth frame of the main camera and the second frame, the third frame, the fourth frame and the fifth frame of the auxiliary camera is greater than the preset value, respectively, and the second frame, the third frame, the fourth frame and the fifth frame of the main camera are not synchronized with the second frame, the third frame, the fourth frame and the fifth frame of the auxiliary camera, respectively. Figure 6A The case in FIG. 7 determines that the cache does not store images of the second camera synchronized with the key frame.

[0094] In some possible implementation, the depth information of the key frame is reconstructed according to one or more pairs of synchronized images in the cache. Specifically, the one or more pairs of images can be the previous images of the key frame, or the one or more pairs of images can be the subsequent images of the key frame, or the one or more pairs of images can be the previous images and the subsequent images of the key frame. For example, for the scene shown in FIG. 6, since the fourth frame is the key frame, the depth information of the key frame can be reconstructed according to the previous synchronized frame (the image pair corresponding to the fourth frame) and the subsequent synchronized frame (the image pair corresponding to the fifth frame) of the fourth frame in the cache. For the scene shown in FIG. 7, since the fourth frame is the key frame, the depth information of the key frame can be reconstructed according to the previous synchronized frame (the image pair corresponding to the first frame) of the fourth frame in the cache. Figure 5A Figure 6A For the scene shown in FIG. 7, since the fourth frame is the key frame, the depth information of the key frame can be reconstructed according to the previous synchronized frame (the image pair corresponding to the first frame) of the fourth frame in the cache.

[0095] 405. generating a blurred snapshot image according to the key frame and the depth information of the key frame.

[0096] With this embodiment, when the depth information of the key frame cannot be obtained according to the preset method, the depth information of the key frame can be determined according to the contour information of the key frame, the depth information, the confidence information and the motion estimation information corresponding to the one or more pairs of synchronized images stored in the cache, and the electronic device can obtain depth information with better quality, which is beneficial to improve the display effect of the blurred snapshot image generated at the snapshot time.

[0097] As an example, refer to Figure 7 ​A structural schematic diagram of an electronic device 700 is provided in embodiments of the present application. As shown in the figure, the electronic device 700 can include a processor 701, a communication module 702, a display screen 703, and the like. Figure 7

[0098] The processor 701 can include one or more processing units, and different processing units can be independent devices or integrated in one or more processors 701. The controller can be the nerve center and command center of the electronic device 700. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of fetching and executing instructions. A memory can also be provided in the processor 701 to store instructions and data.

[0099] In some embodiments, the processor 701 can include one or more interfaces. The interfaces can include an integrated circuit interface, an integrated circuit built-in audio interface, a pulse code modulation interface, a universal asynchronous receiver transmitter interface, a mobile industry processor interface, a general-purpose input / output interface, a subscriber identity module interface, and / or a universal serial bus interface 711, and the like.

[0100] The electronic device 700 can realize display functions through a GPU, the display screen 703, and the application processor 701, and the like. The GPU is a microprocessor for image processing, connected to the display screen 703 and the application processor 701. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 701 can include one or more GPUs that execute program instructions to generate or change display information. The display screen 703 is used to display images, video streams, and the like. The communication module 702 can include an antenna x, an antenna y, a mobile communication module 702A, and / or a wireless communication module 702B. For example, the communication module 702 includes the antenna x, the antenna y, the mobile communication module 702A, and the wireless communication module 702B. In addition, the receiving module provided in embodiments of the present application can also be arranged in the communication module 702.

[0101] The wireless communication function of the electronic device 700 can be realized through the antenna x, the antenna y, the mobile communication module 702A, the wireless communication module 702B, a modem processor, and a baseband processor, and the like.

[0102] ​The mobile communication module 702A can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the electronic device 700. The mobile communication module 702A can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 702A can receive an electromagnetic wave by the antenna x, and perform filtering, amplification, etc. on the received electromagnetic wave, and transfer the processed signal to the modem processor to be demodulated. The mobile communication module 702A can also amplify a signal modulated by the modem processor, and radiate the signal as an electromagnetic wave through the antenna x. In some embodiments, at least part of the functional modules of the mobile communication module 702A can be disposed in the processor 701. In some embodiments, at least part of the functional modules of the mobile communication module 702A can be disposed in the same device as at least part of the modules of the processor 701.

[0103] The application processor outputs a sound signal through an audio device (not limited to the speaker 706A, the microphone 706B, etc.), or displays an image or a video stream through the display screen 703. In some embodiments, the modem processor can be a separate device. In other embodiments, the modem processor can be independent of the processor 701, and disposed in the same device as the mobile communication module 702A or other functional modules.

[0104] The wireless communication module 702B can provide a solution for wireless communication applied to the electronic device 700. The wireless communication module 702B can be one or more devices that integrate at least one communication processing module. The wireless communication module 702B receives an electromagnetic wave through the antenna y, performs frequency modulation and filtering on the electromagnetic wave signal, and transmits the processed signal to the processor 701. The wireless communication module 702B can also receive a signal to be transmitted from the processor 701, perform frequency modulation and amplification, and radiate the signal as an electromagnetic wave through the antenna y.

[0105] The antenna x of the electronic device 700 and the mobile communication module 702A are coupled, and the antenna y and the wireless communication module 702B are coupled, so that the electronic device 700 can communicate with a network and other devices through a wireless communication technology.

[0106] The electronic device 700 can further include an external memory interface 710, an internal memory 704, a universal serial bus (USB) interface 711, a charging management module 712, a power management module 713, a battery 714, an audio module 706, a speaker 706A, a receiver 706B, a microphone 706C, an earphone interface 706D, a sensor module 705, a key 709, a motor, an indicator 708, a camera 707, a subscriber identification module (SIM) card interface, and the like. The electronic device 700 can implement an audio function through the audio module 706, the speaker 706A, the receiver 706B, the microphone 706C, the earphone interface 706D, the application processor 701, and the like. For example, music play, sound recording, and the like. The sensor module 705 in the electronic device 700 can implement sensing and / or acquisition functions for different signals.

[0107] The electronic device provided by the embodiments of the present application is introduced above. It should be understood that the structure illustrated by the embodiments does not constitute a specific limitation on the electronic device 700. In other embodiments, the electronic device 700 can include more or fewer components than illustrated, or combine certain components, or split certain components, or different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0108] Among them, the electronic device, computer readable storage medium, computer program product or chip provided by the embodiments of the present application are used to execute the corresponding method provided above, so the beneficial effects they can achieve can refer to the beneficial effects in the corresponding method provided above, which will not be repeated here.

[0109] The hardware system of the electronic device 700 is described in detail above, and the software system of the electronic device 700 is introduced below.

[0110] Figure 8 is a schematic diagram of the software system of the electronic device provided by the embodiments of the present application.

[0111] As Figure 8 indicated, the system architecture can include an application layer 810, an application framework layer 820, a hardware abstraction layer 830, a driver layer 840, and a hardware layer 850.

[0112] The application layer 810 can include a camera application program.

[0113] Optionally, the application layer 810 can also include a camera application program, a calendar, a call, a map, a navigation, a WLAN, a Bluetooth, a music, a video, a short message, and the like.

[0114] The application framework layer 820 provides an application programming interface (API) and a programming framework for the application programs of the application program layer; the application framework layer can include some predefined functions.

[0115] For example, the application framework layer 820 can include a camera access interface; the camera access interface can include a camera management and a camera device. The camera management can be used to provide an access interface for managing the camera; and the camera device can be used to provide an interface for accessing the camera.

[0116] The hardware abstraction layer 830 is used to abstract the hardware. For example, the hardware abstraction layer can include a camera abstraction layer and other hardware device abstraction layers; the camera abstraction layer can include a camera device 1, a camera device 2, etc.; the camera hardware abstraction layer can be connected to a camera algorithm library, and the camera hardware abstraction layer can call the algorithms in the camera algorithm library.

[0117] The driver layer 840 is used to provide drivers for different hardware devices. For example, the driver layer can include a camera device driver.

[0118] The hardware layer 850 can include an image sensor, an image signal processor (ISP), and other hardware devices; the sensor can be used to collect a Raw image; the Raw image collected by the sensor can be sent to the image signal processor for image processing, and the processed image is output; and the processed image is transmitted to the camera application program of the application layer; the image signal processor can include an automatic exposure module, which is used to perform automatic exposure processing; the automatic exposure module includes a control module, which is used to execute the related algorithms of the generation method of the snapshot image provided in the embodiments of the present application: in the blurring mode, the image frame recently collected by the camera is saved to the cache; a snapshot instruction is obtained; in response to the snapshot instruction, a key frame is determined from the cache; when the depth information of the key frame cannot be obtained according to a preset method, the depth information of the key frame is reconstructed according to the image saved in the cache; and a blurred snapshot image is generated according to the key frame and the depth information of the key frame.

[0119] The present application also provides a computer program product, which, when executed by a processor, implements the interaction method of any method embodiment of the present application.

[0120] The present application also provides a computer readable storage medium, which includes instructions, when the instructions are run on an electronic device, cause the electronic device to execute the interaction method of any method embodiment. The specific steps of the interaction method are described in the foregoing method embodiments, which will not be repeated here.

[0121] It should be noted that the embodiments of the present application are described by taking the Android system as an example, but the basic principles are also applicable to electronic devices based on iOS, Windows and other operating systems.

[0122] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the electronic device, the storage medium and the program product described above can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0124] In several embodiments provided in the present application, it should be understood that the disclosed method, electronic device, storage medium and program product can be implemented in other ways. For example, the embodiments of the electronic device described above are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0125] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0126] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0127] It should be understood that in various embodiments of the present application, the size of the serial number of each process does not mean the execution order, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0128] In addition, the term "and / or" herein only describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0129] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts of the present application that essentially contribute to the prior art or the parts of the technical solutions of the present application can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), and various program code storage media.

[0130] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims. In summary, the above is only a preferred embodiment of the technical solutions of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method of generating a snapshot image, characterized by, The method is applied to an electronic device, and a camera of the electronic device includes a first camera and a second camera, and the method includes: In a blurring mode, saving image frames collected by the first camera and the second camera to a cache; Performing double-camera synchronous detection and confirmation on the image frames collected by the first camera and the image frames collected by the second camera in the cache by a double-camera synchronous detection module to obtain a synchronous image pair in the cache; Obtaining a snapshot instruction; In response to the snapshot instruction, determining a key frame from the cache; the key frame is an image collected by the first camera; Determining depth information and confidence corresponding to one or more image pairs in the synchronous image pair in the cache; Generating a depth quality score of the one or more image pairs according to the confidence; the depth quality score represents whether the depth information is available; In a case where no image collected by the second camera synchronized with the key frame is saved in the cache, or the depth quality score of the key frame is less than a preset threshold, according to contour information of a target object, depth information, confidence information and motion estimation information corresponding to the one or more image pairs relative to the key frame, the depth information of the key frame is reconstructed; the contour information is determined according to the key frame; Generating a blurred snapshot image according to the key frame and the depth information of the key frame.

2. The method of claim 1, wherein, The method further includes: If the image collected by the second camera synchronized with the key frame is saved in the cache, and the depth quality score is greater than or equal to the preset threshold, the depth information of the key frame is not reconstructed according to the contour information, the depth information corresponding to the one or more image pairs, the confidence information and the motion estimation information.

3. The method of claim 1, wherein, Before the depth information of the key frame is reconstructed according to the contour information of the target object, the depth information corresponding to the one or more image pairs, the confidence information and the motion estimation information relative to the key frame, the method further includes: Determining the motion estimation information of the one or more image pairs relative to the key frame.

4. The method of claim 1, wherein, Before the depth information of the key frame is reconstructed according to the contour information of the target object, the depth information corresponding to the one or more image pairs, the confidence information and the motion estimation information relative to the key frame, the method further includes: A human segmentation module determines the contour information of the target object according to the key frame.

5. The method of any one of claims 1-4, wherein: The one or more image pairs are previous images of the key frame; or The one or more image pairs are subsequent images of the key frame; or The multiple image pairs are the previous images and the subsequent images of the key frame.

6. An electronic device, comprising: A memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method of any one of claims 1-5.

7. A computer readable storage medium characterized in that, The computer readable storage medium includes a stored program, wherein the program, when executed, controls a device in which the computer readable storage medium is located to perform the method of any one of claims 1-5.

8. A computer program product, characterised in that, The computer program product contains executable instructions, which, when executed on a computer, cause the computer to perform the method of any one of claims 1-5.

Images