Photographing method
By determining the best frame of the image in the electronic device and updating the photos, the problem of difficult to capture wonderful photos caused by reaction delay and transmission delay during the shooting process is solved, and the shooting experience and photo quality is improved.
Patent Information
- Application Number
- PCT/CN2023/136216
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-02-27
- Filing Date
- 2023-12-04
- Publication Date
- 2024-11-14
AI Technical Summary
During the shooting process, due to the human reaction delay and the transmission delay of electronic devices, it is difficult for users to take wonderful photos that meet expectations, resulting in a reduced shooting experience.
By implementing a shooting method in an electronic device, the optimal frame of the image is determined using a fixed step time window, and the photos are updated according to the comprehensive score level, ensuring the output photo effect and quality. The method includes generating a photo and thumbnail when a shooting operation is detected, and continuing to acquire images after the shooting operation to update the photos, ensuring that the user has a more exciting photo experience.
It improves the user's shooting experience, ensures the quality and effect of the output photos, reflects the user's initiative in capturing, and reduces the waste of storage resources.
Smart Images

Figure CN2023136216_14112024_PF_FP_ABST
Abstract
Description
A shooting method
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 27, 2023, with application number 202310216815.2 and application name “A Shooting Method”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of terminals, and in particular to a shooting method. Background Art
[0003] During the shooting process, due to issues such as human reaction delay and electronic device transmission delay, users often find it difficult to take the wonderful photos they expect, which greatly reduces the user's shooting experience.
[0004] Summary of the Invention
[0005] This application provides a shooting method. By implementing this shooting method, the electronic device 100 can determine the most exciting image from images taken within a period of time, including the moment of the shooting operation, and then output it as a photo. This method reflects the user's initiative in capturing photos, while ensuring the effect and quality of the final output photo, providing the user with a better shooting experience.
[0006] In a first aspect, the present application provides a shooting method, which is applied to an electronic device, the electronic device including a camera, the method including: opening a camera application; at time T1, displaying a preview interface, the preview interface including a preview area and a thumbnail area, the preview area displaying an image captured by the camera in real time; detecting a shooting operation at time T2; in response to the shooting operation, the thumbnail area displays a thumbnail of a first photo, the first photo is obtained based on the first image, the first image is an image captured by the camera after time T1 and before time T2; in response to the shooting operation, the thumbnail area displays a thumbnail of a second photo, the second photo is obtained based on the second image, the second image is an image captured by the camera after time T2 and excluding time T2.
[0007] When implementing the method provided in the first aspect, upon detecting a capture operation, the electronic device may first generate a photo and thumbnail based on the frame of imagery preceding the capture operation, which serve as the photo and thumbnail resulting from the user's capture operation. The electronic device may then continue to capture images subsequent to the capture operation and generate a photo and thumbnail based on the frame of imagery following the capture operation, updating the photo and thumbnail resulting from the user's capture operation. This allows the user to enjoy a unique photography experience.
[0008] In combination with the method provided in the first aspect, in some embodiments, before the thumbnail area displays the thumbnail of the second photo, the method also includes: detecting a first operation on the thumbnail of the first photo; in response to the first operation, displaying a second interface, the second interface including a third area, and the first photo being displayed in the third area; replacing the first photo displayed in the third area with the second photo.
[0009] By implementing the method provided in the above embodiment, after generating a photo and a thumbnail based on a frame of image before the shooting operation and detecting the user operation of browsing the above photo, the electronic device can update the taken photo during the browsing of the taken photo.
[0010] In combination with the method provided in the first aspect, in some embodiments, the method also includes: opening a gallery application, displaying a third interface, the third interface displaying one or more images, the one or more images including a preview thumbnail of the second photo, but not including a preview thumbnail of the first photo.
[0011] By implementing the method provided in the above embodiment, the electronic device can only save the final photo, such as the second photo, to reduce storage resource waste. When the user browses the gallery, only the final photo is displayed.
[0012] In combination with the method provided in the first aspect, in some embodiments, the preview interface further includes a second control, and the method further includes: detecting a user operation acting on the second control, and enabling a manual snapshot function.
[0013] When implementing the method provided in the above embodiment, the electronic device can set a feature switch, such as the second control described above, to control the activation of manual snapshot. When a user requests a manual snapshot, the electronic device can activate manual snapshot based on the operation of the feature switch and then perform the subsequent operations of identifying, capturing, and saving outstanding images. This allows the electronic device to save power.
[0014] In combination with the method provided in the first aspect, in some embodiments, the first image is a frame of image with the highest comprehensive score captured by the camera after time T1 and before time T2, and the comprehensive score of the image is determined by one or more of the following: the number, size and / or position of the first object, the completion of the first action, the number, size and / or position of the face, the degree of smile of the face, the degree of eye opening of the face, clarity, or exposure; the second image is a frame of image with the highest comprehensive score captured by the camera after time T2 and excluding time T2, and the comprehensive score of the second image is higher than the comprehensive score of the first image.
[0015] By implementing the method provided in the above embodiment, the electronic device can determine the best frame within a period of time before and after a capture operation using one or more of the following methods. Upon detecting a capture operation, the electronic device can first generate a photo and thumbnail based on the best frame before the capture operation, which serve as the photo and thumbnail resulting from the user's capture operation. The electronic device can then continue to capture images after the capture operation and generate a photo and thumbnail based on the best frame after the capture operation, updating the photo and thumbnail resulting from the user's capture operation.
[0016] In this way, the electronic device can not only generate a wonderful photo in time according to the determined best frame when the shooting operation occurs, but also update the taken photo after detecting a more wonderful image in the future, so that the user can finally take a more wonderful photo.
[0017] In combination with the method provided in the first aspect, in some embodiments, the comprehensive score of the second image is higher than the comprehensive score of the first image, and the difference between the comprehensive score of the second image and the comprehensive score of the first image is higher than a first preset value.
[0018] By implementing the method provided in the above embodiment, the electronic device 100 can prioritize the image before the capture operation, thereby ensuring that the final captured photo better meets the user's capture intent. Only when the best frame after the capture operation is significantly better than the best frame before the capture operation will the electronic device 100 update the captured photo to the best frame after the capture operation.
[0019] In combination with the method provided in the first aspect, in some embodiments, before the shooting operation is detected at time T2, the method further includes: selecting the first best frame with the highest comprehensive score from the images captured by the camera from time T3 to time T4; time T3 and time T4 are after time T1 and before time T2, and time T3 and time T4 are after time T1, and are determined according to a time window with a fixed step size; in response to the shooting operation, the thumbnail area displays a thumbnail of the first photo, and the first photo is obtained based on the first image, specifically including: in response to the shooting operation, selecting the second best frame with the highest comprehensive score from the images captured by the camera from time T4 to time T2; determining the one with a higher comprehensive score from the first best frame and the second best frame as the first candidate frame; the first image is the first candidate frame.
[0020] By implementing the method provided in the above embodiment, the electronic device can evaluate the images captured by the camera according to a time window with a fixed step size after entering the camera application (in the scenario where manual capture is turned on), and determine the best frame for each window. After detecting the shooting operation, another best frame is determined based on the existing image in the current window, and then the better one is selected from the two most recent best ones as the first candidate frame. The electronic device can then output the first candidate frame that performed best before the shooting operation, such as the first image, as a photo, as a temporary photo taken according to the user's manual shooting operation.
[0021] By implementing the method provided in the above embodiment, the electronic device can immediately determine the best performing image before the shooting operation after detecting the shooting operation, thereby obtaining temporary photos and thumbnails as quickly as possible, giving the user more timely shooting feedback.
[0022] In combination with the method provided in the first aspect, in some embodiments, in response to a shooting operation, the thumbnail area displays a thumbnail of the second photo, and the second photo is obtained based on the second image, specifically including: selecting the second candidate frame with the highest comprehensive score from the images captured by the camera from time T2 to time T5; determining that the comprehensive score of the second candidate frame is higher than that of the first candidate frame, and the second image is the second candidate frame.
[0023] By implementing the method provided in the above embodiment, the electronic device can continue to capture images for a period of time after the capture operation and, using the same method, determine the best frame within that period, i.e., the second candidate frame. The electronic device can then select the more compelling image from the first candidate frame before the capture operation and the second candidate frame after the capture operation as the final photo captured by the user. This approach not only demonstrates the user's initiative in capturing photos, but also ensures the quality and quality of the final output photo, providing a better user photography experience.
[0024] In combination with the method provided in the first aspect, in some embodiments, the time length corresponding to time T2 to time T5 is the same as or different from the time length corresponding to time T3 to time T4.
[0025] In combination with the method provided in the first aspect, in some embodiments, in response to a shooting operation, before the thumbnail area displays a thumbnail of the first photo, the method also includes: determining that the third image includes the first object, and the third image is the image displayed in the preview area when the electronic device detects the shooting operation.
[0026] When implementing the method provided in the above embodiment, the electronic device may first determine, based on the image at the time of the capture operation, whether the target capture scene is a preset typical scene that includes a specific object. If the image at the time of the capture operation includes the specific object, that is, if the image at the time of the capture operation is an image of a typical scene, the electronic device may capture and output the image according to the method provided in the first aspect and any of the above embodiments of the first aspect.
[0027] In combination with the method provided in the first aspect, in some embodiments, the method also includes: when the third image does not include the first object, displaying a thumbnail of the fourth photo in the thumbnail area, the fourth photo is obtained based on the fourth image, the fourth image is the convergence image closest to the third image, and the convergence image includes satisfying one or more of the following: autofocus convergence, auto-exposure convergence, or auto-white balance convergence.
[0028] By implementing the method provided in the above embodiment, when the image at the moment of the shooting operation does not include a specific object, that is, when the image is in an atypical scene, the electronic device does not need to care about the image content and can obtain a convergent image with higher image quality at a lower cost, thereby ensuring the timeliness of the photos taken, while also reducing computing costs and saving power consumption.
[0029] In combination with the method provided in the first aspect, in some embodiments, the first photo is obtained based on the first image, specifically including: obtaining multiple frames of images before and after including the first image captured by the camera; fusing the multiple frames of images into a new frame of image, and the first photo is obtained based on the new image.
[0030] By implementing the method provided in the above embodiment, the electronic device can process the wonderful image in combination with other images near the best frame (multi-frame fusion), thereby further improving the quality of the wonderful image, such as color, brightness, saturation, etc.
[0031] In combination with the method provided in the first aspect, in some embodiments, before acquiring multiple frames of images including the first image captured by the camera, the method further includes: starting a first shooting mode.
[0032] By implementing the methods provided in the above embodiments, the electronic device can determine whether to perform multi-frame fusion based on the shooting mode. In certain specific shooting modes, the electronic device can significantly improve the quality of the final output photo through multi-frame fusion. In other scenarios, the quality improvement brought by multi-frame fusion is not significant. In this case, the electronic device may not perform multi-frame fusion to save resources and reduce power consumption.
[0033] In combination with the method provided in the first aspect, in some embodiments, the first shooting mode is a night scene mode.
[0034] By implementing the method provided in the above embodiment, when shooting in night scene mode, the electronic device can perform multi-frame fusion to significantly improve the quality of the final output photo.
[0035] In a second aspect, the present application provides an electronic device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code comprises computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation of the first aspect.
[0036] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the method described in the first aspect and any possible implementation method of the first aspect.
[0037] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.
[0038] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.
[0039] It is understandable that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to perform the methods provided in this application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] FIG1 is a schematic diagram of setting a window and determining an optimal frame according to an embodiment of the present application;
[0041] FIG2 is a schematic diagram of implementing manual snapshot according to an embodiment of the present application;
[0042] FIG3A is a diagram of a software architecture for implementing manual snapshots provided in an embodiment of the present application;
[0043] FIG3B is a schematic diagram of a method for determining image brilliance provided by an embodiment of the present application;
[0044] 4A-4C are a set of user interfaces for enabling manual snapshot according to an embodiment of the present application;
[0045] 5A-5H are a set of user interfaces for implementing manual snapshots provided in an embodiment of the present application;
[0046] 6A-6D are another set of user interfaces for implementing manual snapshots provided in an embodiment of the present application;
[0047] 7A-7J are another set of user interfaces for implementing manual snapshots provided in an embodiment of the present application;
[0048] FIG8 is a flowchart of obtaining the best frame of a window provided by an embodiment of the present application;
[0049] FIG9 is a flowchart of implementing manual snapshot according to an embodiment of the present application;
[0050] Figures 10A-10E are a set of user interfaces for enabling the automatic snapshot function provided in an embodiment of the present application;
[0051] 11A-11H are a set of user interfaces for implementing automatic snapshots provided in an embodiment of the present application;
[0052] FIG12 is a schematic diagram of the system structure of the electronic device 100 provided in an embodiment of the present application;
[0053] FIG13 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application. DETAILED DESCRIPTION
[0054] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.
[0055] During the shooting process, due to the delay in human reaction, users often find it difficult to take wonderful photos that meet their expectations.
[0056] For example, at time T1, the kth frame image may be displayed in the preview window of the electronic device 100. The electronic device 100 is an electronic device equipped with a camera and providing shooting and display services, such as a smart phone, a tablet computer, etc.
[0057] After the user sees the above-mentioned k-th frame image through the preview window of the electronic device 100, he or she may determine that this frame image is very exciting. At this time, the user may determine to perform a shooting operation to capture the k-th frame image. The image that the user wants to capture, such as the above-mentioned k-th frame image, may be referred to as the target image. However, due to human reaction delay (including the time required to complete the shooting operation), the moment when the electronic device 100 receives the user's shooting operation will lag, resulting in the image that is finally matched after the target image, such as the k+n-th frame image, rather than the k-th frame image that the user expects to capture.
[0058] In particular, due to the uncertainty of human reaction delay and the device jitter caused by human movement during the click-to-shoot action, the k+n frame image is more likely to have ghosting, which greatly affects the image quality of the final photo and the user's shooting experience.
[0059] The electronic device 100 is not limited to smartphones and tablet computers. It can also be a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a wearable device, an in-vehicle device, a smart home device and / or a smart city device. The embodiments of the present application do not impose any special restrictions on the specific type of the electronic device 100.
[0060] In view of this, an embodiment of the present application provides a shooting method.
[0061] When implementing the shooting method provided in the embodiments of the present application, after turning on the camera, the electronic device 100 can determine a time window containing a fixed number of image sequences at a fixed step size. For any window, the electronic device 100 can determine a frame with the best overall evaluation of image content and image quality from all images in the window, and record this as the best frame.
[0062] Among them, the indicators for evaluating the image content include but are not limited to: including specific objects, the size and position of the objects, including specific actions, the degree of completion of specific actions, including human faces, the size and position of human faces, and facial expressions (whether eyes are open, whether smiling, etc.). The above-mentioned specific objects (first objects) are, for example, people, cats, dogs, etc. The above-mentioned specific actions are, for example, jumping, looking back, parachuting, etc. The embodiments of this application do not limit the above-mentioned specific objects and specific actions. The indicators for evaluating image quality include, but are not limited to, clarity, exposure, etc.
[0063] When detecting a user's shooting operation, the electronic device 100 may first determine an optimal frame from the images already obtained in the current window. Subsequently, the electronic device 100 may select the frame with the better overall evaluation from the two most recently determined optimal frames and record it as the first candidate frame. Simultaneously, the electronic device 100 may continue to acquire images for a period of time after the shooting operation and determine the best frame for a period of time after the shooting operation using the same method, recording it as the second candidate frame. Finally, the electronic device 100 may select the frame with the better overall evaluation from the first and second candidate frames and output it as a photo.
[0064] This method of selecting the best-performing image from a period of time, including the moment of the capture, based on the user's shooting action, can be called manual capture. This approach not only demonstrates user initiative but also ensures the quality of the final output photo, providing a better user shooting experience.
[0065] FIG1 is a schematic diagram of setting a window and determining an optimal frame according to an embodiment of the present application.
[0066] After manual capture is enabled, the electronic device 100 may determine a time window containing a fixed image sequence according to a fixed step size. For example, the fixed step size may be set to 15 (frames), i.e., the time range corresponding to every 15 consecutive frames of images may be referred to as a time window, or simply a window.
[0067] As shown in FIG1 , the time range corresponding to the 1st to 15th frames of images may be referred to as Window 1; the time range corresponding to the 16th to 30th frames of images may be referred to as Window 2; the time range corresponding to the 31st to 45th frames of images may be referred to as Window 3; and the time range corresponding to the 46th to 60th frames of images may be referred to as Window 4. Similarly, as images are continuously generated, the electronic device 100 may determine more windows, which are not listed here one by one.
[0068] After acquiring any image frame, electronic device 100 may input it into a preset algorithm for obtaining image content and image quality, and obtain a comprehensive score indicating the comprehensive evaluation of the image content and image quality. The process by which electronic device 100 obtains the comprehensive score using the preset algorithm will be described in detail in subsequent embodiments and will not be elaborated here.
[0069] After the comprehensive scores of all images in a window are determined, the electronic device 100 may determine the image with the highest comprehensive score from all images in the window as the best frame in the window based on the comprehensive scores of each frame.
[0070] For example, the 10th frame image in window 1 may be the image with the highest comprehensive score in window 1. Therefore, electronic device 100 may determine the 10th frame image as the best frame for window 1, denoted as K1. Similarly, electronic device 100 may sequentially determine the best frames for windows 2 through 4, denoted as K2 through K4, respectively.
[0071] FIG2 is a schematic diagram of implementing manual snapshot provided in an embodiment of the present application.
[0072] First, when a window ends and no user shooting operation is detected, the electronic device 100 can determine the best frame of the window after the window ends according to the method shown in Figure 1. For example, the electronic device 100 can determine the best frame K1 (the first best frame) in window 1. The start time of window 1 can be referred to as time T3, and the end time of window 1 can be referred to as time T4.
[0073] The user can perform a shooting operation at any time. The electronic device 100 can detect in real time whether the user has performed a shooting operation. In other words, the electronic device 100 can detect the user's shooting operation at any time. For example, as shown in Figure 2, the electronic device 100 can detect the user's shooting operation at time t. Time t is within window 2.
[0074] In response to the above operation, the electronic device 100 may first obtain the best frame of window X from window X, which is recorded as Ks1. Window X is the time range corresponding to the image obtained in the current window 2. Typically, the step size of window X is smaller than the aforementioned preset fixed step size.
[0075] For example, time t (time T2) corresponds to the 24th frame image (the third image), and window X is the time range corresponding to frames 16 to 24. In this case, the electronic device 100 can determine the image with the highest comprehensive score from frames 16 to 24, such as frame 18, as the best frame Ks1 (the second best frame) in window X.
[0076] In this case, K1 and Ks1 are the two best frames most recently determined by the electronic device 100 when the capture operation is detected. The electronic device 100 may then select the frame with the better overall evaluation (i.e., the frame with the higher overall score) from the two best frames as the first candidate frame. For example, as shown in FIG2 , the electronic device 100 may determine the best frame Ks1 with the higher overall score as the first candidate frame. The first candidate frame Ks1 may be referred to as the first image. The time range corresponding to the window 1 and window X may be referred to as the first time range.
[0077] After detecting the shooting operation, the electronic device 100 can continue to acquire images within a period of time after the shooting operation. As shown in Figure 2, the electronic device 100 can continue to acquire images within the time period from t to t'. t-t' can be referred to as window Y. Time t' is also referred to as time T5. The step size of window Y can be the same as or different from the aforementioned preset fixed step size, and this embodiment of the application does not limit this. The time range corresponding to the above-mentioned window Y can be referred to as the second time range.
[0078] Then, the electronic device 100 can refer to the above-described method for determining the best frame to determine the best frame from window Y, which is recorded as Ks2. Ks2 is the second candidate frame for this shooting operation. The second candidate frame Ks2 can be called a second image.
[0079] After determining the first candidate frame Ks1 before the capture operation (including the moment the capture operation was initiated) and the second candidate frame Ks2 after the capture operation, the electronic device 100 may select the image with the higher comprehensive score from the first candidate frame Ks1 and the second candidate frame Ks2 as the wonderful image. For example, the electronic device 100 may determine that Ks2, with the higher comprehensive score, is the wonderful image. Finally, the electronic device 100 may perform a photo processing on the wonderful image Ks2 to obtain the corresponding wonderful photo.
[0080] In some embodiments, when comparing the first candidate frame and the second candidate frame, the electronic device 100 may determine that the second candidate frame is a great image only after determining that the second candidate frame is higher than the first candidate frame and higher than a preset value. The above preset value is recorded as the first preset value. In this way, the electronic device 100 can place greater emphasis on the image before the capture operation, thereby more closely matching the user's capture intent.
[0081] FIG3A is a diagram of a software architecture for implementing manual snapshots according to an embodiment of the present application.
[0082] The electronic device 100 is equipped with a camera. The image obtained after the photosensitive element in the camera converts the captured light source signal into an electrical signal can be called an original image, also known as a RAW image. The above process is also called camera acquisition of the image. The process of the camera sending the acquired image to the functional module in the system structure of the electronic device 100 can be called reporting. The module responsible for shooting and outputting photos (such as the processing and capturing module shown in Figure 3A) processes the image reported by the camera, and then outputs it as a photo or other image resource and saves it. The process can be called shooting.
[0083] In the zero shutter lag (ZSL) framework, a rotating buffer is provided in the electronic device 100. The buffer is used to store a set of RAW images recently reported by the camera. For example, the size of the rotating buffer is 6, that is, the 6 frames of RAW images recently reported by the camera can be stored, recorded as Fi-5~Fi. After the camera reports the i+1th frame RAW image, the rotating buffer will write the i+1th frame RAW image Fi+1 and delete the earliest written image Fi-5 in the buffer. At this time, the images stored in the rotating buffer are updated to Fi-4~Fi+1.
[0084] The smaller image that can be obtained by downsampling the RAW image is called a processed image (also called a Tiny image). The image stream composed of continuous Tiny images is called a Tiny stream. The RAW image used in the above-mentioned process can be directly derived from the RAW image reported by the camera, or directly derived from the RAW image stored in the rotating buffer. The embodiment of the present application does not limit this. For example, the size of the RAW image can be 3024*4032(p), and the size of the downsampled image in the Tiny stream can be 378*502(p).
[0085] In some embodiments, images in a Tiny stream can also be extracted from RAW images reported by the camera or stored in a rotating buffer. For example, if a set of RAW images reported by the camera or stored in a rotating buffer includes frames 1 to 20, the module that performs downsampling to obtain the Tiny stream can extract a frame every three frames from these frames, such as frame 1, frame 5, frame 9, and so on, to generate the corresponding Tiny image.
[0086] The Tiny image and the RAW image are aligned by frame number. For example, the frame number of the RAW image reported by the camera is i. The Tiny image obtained by downsampling the RAW image Fi is the i-th Tiny image, denoted as fi. The frame number of fi is i.
[0087] The electronic device 100 is also provided with dedicated buffers, including dedicated buffer A and dedicated buffer B. The dedicated buffers are used to store the best frame in RAW image format. Specifically, before detecting a capture operation by the user, dedicated buffer A and dedicated buffer B alternately store the most recently determined best frame. After detecting a capture operation by the user, the best frame with a better overall evaluation in dedicated buffer A and dedicated buffer B is retained, while the best frame with a poorer overall evaluation is discarded. Subsequently, the electronic device 100 may write the best frame within a period of time after the capture operation, i.e., the second candidate frame, into the dedicated buffer corresponding to the discarded best frame.
[0088] The electronic device 100 is pre-installed with a perception module and a decision module. The perception module is pre-installed with multiple detection algorithms for acquiring various data indicating image content and image quality. The decision module can determine a comprehensive image score based on this data and, based on the comprehensive score, determine the optimal frame for each image sequence window.
[0089] As shown in Figure 3A, the perception module may include a subject detection algorithm, a motion evaluation algorithm, a face detection algorithm, a face attribute algorithm, and a clarity algorithm. The subject detection algorithm, motion evaluation algorithm, face detection algorithm, and face attribute algorithm can be used to obtain image content, while the clarity algorithm can be used to obtain image clarity. Indicators for evaluating image quality may include other parameters besides clarity, such as exposure. These parameters can be directly obtained from data reported by the camera and / or other sensors. In this case, the perception module may not include an algorithm for extracting these parameters from the image.
[0090] Specifically, the subject detection algorithm is used to obtain the subject in the image, such as a person, a cat's face, a dog, etc. In particular, for subjects other than people, such as the aforementioned cats and dogs, the subject detection algorithm can also obtain partial objects of these subjects, such as cat faces and dog faces. It will be understood that the embodiments of this application do not limit the specific type of subject.
[0091] After identifying a preset subject, the subject detection algorithm can output an object frame. Furthermore, the object frame indicates the size and position of the object in the image. Therefore, the decision module can determine whether the image includes a specific subject based on whether the subject detection algorithm outputs an object frame. Furthermore, the decision module can determine the number, size, and position of the objects based on the number, size, and position of the output object frames, and then evaluate the objects and determine an object score. For example, the larger the object frame and the more centrally located it is, the higher the object score corresponding to that object frame.
[0092] The action evaluation algorithm captures the subject's movements, such as jumping, looking back, throwing objects, playing table tennis, badminton, running, splashing water, playing baseball, playing rugby, flying Frisbee, and so on. It then determines the degree of action completion in the current image based on the optimal completion point for each behavior and outputs an action score for the image. Images with higher action completion levels receive higher scores.
[0093] For example, taking jumping as an example, the motion state where the object reaches its highest point after leaving the ground can be called the optimal completion point for the jump. When the object in the i-th frame image is in a motion state where it leaves the ground and reaches its highest point, the motion evaluation algorithm can output a motion score of 1 (i.e., the highest score) for the image. When the object in the image does not leave the ground, or leaves the ground but does not reach its highest point, the motion evaluation algorithm can output a motion score of 0.3 or 0.5, or a score less than 1. Specifically, the motion evaluation algorithm can determine the motion score at different moments when the object leaves the ground but does not reach its highest point based on the distance from the highest point.
[0094] It is understandable that determining the optimal completion point of an action requires determining the trajectory of the subject over a period of time. Therefore, after receiving multiple frames of images over a period of time (e.g., a time window), the action evaluation algorithm can first determine the optimal completion point in these multiple frames. Then, based on the optimal completion point, it evaluates the motion state of each frame in the multiple frames and outputs an action score for each frame.
[0095] In some embodiments, the action evaluation module can also recognize actions of non-human objects, such as a cat running, jumping, etc.
[0096] The face detection algorithm is used to detect faces in an image. Once a face is recognized, the face detection algorithm outputs a face frame. This frame identifies the size and position of the face in the image. The decision module then determines whether the image contains a face based on whether the face detection algorithm outputs a face frame. Furthermore, the decision module evaluates the face based on the number, size, and position of the output face frames, outputting a face score. For example, a larger face frame and a more centered face frame will be associated with a higher face score.
[0097] The facial attribute algorithm is used to obtain facial information from the image, such as the curvature of the lips and the distance between the upper and lower eyelids. The decision module can use this information to determine whether the face is smiling, whether the eyes are open, and so on. Furthermore, the decision module can determine the degree of smile based on the curvature of the lips and the degree of eye openness based on the distance between the upper and lower eyelids. The decision module can then evaluate the facial performance based on the degree of smile and eye openness, outputting a facial performance score.
[0098] In one implementation, a larger lip curvature corresponds to a higher degree of smiling, and thus, a higher facial performance score determined by the decision module. In another implementation, a larger distance between the upper and lower eyelids corresponds to a higher degree of eye openness, and thus, a higher facial performance score determined by the decision module. When combining these two factors, the weights of eye openness and smiling on the facial performance score can be equal or different.
[0099] In an optional implementation, face detection can be performed after subject detection identifies a person. Similarly, the facial attribute algorithm can be performed after the face detection algorithm identifies a face. In this case, the face detection algorithm can use the image containing the person identified by subject detection to obtain the face, and the facial attribute algorithm can use the face image output by the face detection algorithm to obtain the facial information of each face, thereby saving computing costs and improving algorithm efficiency.
[0100] The clarity algorithm is used to obtain image clarity. The clearer the image is, the clearer it is.
[0101] It is understood that the perception module can also use more detection algorithms to obtain more data indicating image content and image quality. This embodiment of the present application is not limited to this. Alternatively, the perception module can also directly obtain information indicating image content and image quality, such as exposure, from data reported by the camera and / or other sensors. This embodiment of the present application is not limited to this.
[0102] The decision module determines a comprehensive score for each image frame based on the image content and quality data output by the perception module. A higher comprehensive score indicates a more compelling image. After determining the comprehensive score for each frame, the decision module can then determine the optimal frame for that window based on the comprehensive scores of each image within that window.
[0103] FIG3B is a schematic diagram of a method for determining a comprehensive image score provided in this application.
[0104] As shown in Figure 3B, the perception module may include a single-frame perception module and a multi-frame perception module. The single-frame perception module can use a frame of image to determine a type of information about the frame, such as whether it contains a specific object, whether it contains a human face, facial attributes, clarity, exposure, etc. The single-frame perception module is suitable for obtaining a type of information from one or more frames of a group of images, such as an action score.
[0105] In one implementation, the decision module may combine multiple categories of information output by the perception module according to image content and image quality, single frame and multiple frames, reduce the number of items, and then determine the final comprehensive score based on the scores of the reduced items.
[0106] As shown in Figure 3B, the decision module can fuse the image quality information, such as clarity and exposure, output by the quality perception module to generate a quality score indicating image quality. This fusion can be performed, for example, using a weighted average. The decision module can also fuse the object boxes, face boxes, and facial attributes output by the content perception module to generate a static content score indicating the content of the static image. In this case, the decision module can obtain scores for three categories: a quality score, a static content score, and a motion score (i.e., a dynamic content score).
[0107] The decision module may determine the static content score based on one or more of the object frame, face frame, and facial attributes output by the content perception module. For example, the decision module may first determine the subject in the image. When no specific object is identified, the results output by the face detection algorithm and the face attribute algorithm are correspondingly null. In this case, the decision module may determine the static content score to be 0 based on the fact that the subject detection module does not output a valid object frame. When a face is identified, the decision module may determine the static content score based on the object frame output by the subject detection module, the face frame output by the face detection module, and / or the facial attributes (eyes open / smiling).
[0108] When the subject detection algorithm identifies that the image does not contain a person, for example, when the subject is a cat or dog, the corresponding output of the face detection algorithm and the face attribute algorithm is empty. In this case, the decision module can determine the static content score based on the object box output by the subject detection module, without considering the output of the face detection algorithm and the face attribute algorithm. This prevents the empty value of the face detection algorithm and the face attribute algorithm from lowering the overall score of the image. This is not listed here one by one.
[0109] After determining the quality score, the static content score, and the action score, the decision module may determine a comprehensive score according to one or more of the quality score, the static content score, and the action score.
[0110] For example, when an image does not include any specific objects or specific actions, the static content score and the action score are both 0 or invalid values. In this case, the decision module may determine the comprehensive score of the image based solely on the quality score, thereby avoiding the situation where the static content score and the action score are 0 or invalid values and the comprehensive score of the image is lowered. For another example, when an image does not include any specific objects but does not include any specific actions, the action scores are both 0 or invalid values. In this case, the decision module may determine the comprehensive score of the image based solely on the quality score and the static content score, thereby avoiding the situation where the action score is 0 or invalid values and the comprehensive score of the image is lowered.
[0111] Due to the differences between content-aware algorithms (subject detection algorithms, face detection algorithms, and face attribute algorithms) and action detection algorithms, in some embodiments, the action evaluation module can determine that a group of images includes a specific action and determine the action score for each frame based on the completeness of the action in each image. However, the content-aware algorithm cannot identify specific objects or their static representations. In this case, the decision module can also determine the overall score of the image based on the quality score and the action score, thereby avoiding the situation where the static content score is 0 or invalid, which would lower the overall score of the image.
[0112] The embodiment of the present application does not limit the specific method of determining the comprehensive image score (excitement) based on the output results of the perception module.
[0113] The electronic device 100 also includes a hardware abstraction layer (HAL). The HAL may include a processing cache module and a processing snapshot module. The processing cache module is used to obtain the best frame in RAW format and write it to a dedicated buffer. The processing snapshot module is used to determine the best image and output the photo.
[0114] Taking the i-th image frame as an example, the downsampled i-th Tiny image fi is first input into the perception module. After processing by the various detection algorithms in the perception module and data reported by the camera and / or other sensors, the decision module obtains various data indicating the content and quality of the image fi, including object bounding boxes, motion scores, face bounding boxes, facial expression scores, clarity, and exposure. Then, referring to Figure 3B , the decision module determines a comprehensive score for fi based on this data.
[0115] After the comprehensive scores of all images in the window corresponding to fi are determined, the decision module can report the event of finding the best frame to the HAL. The above event can include the frame number of the best frame.
[0116] For example, fi is the image with frame number 10 (i=10) in window 1. Window 1 also includes frames f1 to f9 and f11 to f15. For example, after determining the comprehensive score of f15, electronic device 100 may determine that f10 has the highest comprehensive score among frames f1 to f15. In this case, electronic device 100 may determine that f10 is the best frame (K1) in window 1. The decision module may then report the event of finding the best frame to the HAL. In this case, the event includes frame number 10.
[0117] After receiving the reporting event from the decision module and obtaining the frame number, the processing cache module can obtain the RAW image corresponding to the frame number from the rotating buffer and then write it into the dedicated buffer.
[0118] For example, after receiving frame number 10, the processing cache module can obtain the RAW image F10 (i.e., RAW format K1) corresponding to frame number 10 from the rotating buffer and then write F10 to the dedicated buffer. Initially, both dedicated buffer A and dedicated buffer B are empty, and the processing cache module can write F10 to either dedicated buffer. For example, the processing cache module can write F10 to dedicated buffer A.
[0119] Similarly, after determining the best frame K2 for window 2, electronic device 100 may report a best frame discovery event to the HAL. This event may include the frame number of best frame K2, for example, i = 20. Subsequently, the processing cache module may retrieve the RAW image F20 (i.e., RAW format K2) corresponding to frame number 20 from the rotating buffer. The processing cache module may then write F20 to another dedicated buffer: dedicated buffer B.
[0120] After receiving the frame number (e.g., i=35) of the best frame K3 of window 3 reported by electronic device 100, the processing cache module can retrieve the corresponding RAW image F35 from the rotating buffer. At this point, the processing cache module can write F35 to dedicated buffer A, replacing the best frame F10 of window 1 stored earlier. After receiving the frame number (e.g., i=50) of the best frame K4 of window 4 reported by electronic device 100, the processing cache module can retrieve the corresponding RAW image F50 from the rotating buffer. At this point, the processing cache module can write F50 to dedicated buffer B, replacing the best frame F20 of window 2 stored earlier.
[0121] In this way, the dedicated buffer A and the dedicated buffer B receive the latest best frame alternately, so that the two latest best frames are always cached.
[0122] At any time, after the electronic device 100 detects the user's shooting operation, the decision module can determine the latest best frame based on the image obtained in the current window and report it to the HAL. Refer to the window X and the best frame Ks1 shown in Figure 2, which will not be described here. Similarly, the processing cache module in the HAL can write the RAW format image of the above-mentioned latest best frame into an empty buffer area or a dedicated buffer for caching earlier best frames. For example, if K1 is cached in dedicated buffer A, the processing cache module can write Ks1 into dedicated buffer B.
[0123] At the same time, the decision module may also report to the HAL the indication information of the detection of the shooting operation. After receiving the above indication information, the processing and snapshot module may trigger the snapshot. Specifically, the processing and snapshot module may determine the best frame with a higher comprehensive score from the two latest best frames cached in the dedicated buffer as a candidate frame. For example, the processing and snapshot module may determine Ks1 with a higher comprehensive score as a candidate frame (first candidate frame) from K1 cached in the dedicated buffer A and Ks1 cached in the dedicated buffer B. Subsequently, the processing and snapshot module may process the first candidate frame Ks1 and output a photo. In the process of outputting the photo, the processing and snapshot module may simultaneously generate a corresponding thumbnail for preview. At this time, the photo and thumbnail obtained based on the first candidate frame Ks1 are temporary.
[0124] After reporting the most recent best frame (Ks1) and indication information before the capture operation to the HAL, the decision module continues to acquire images for a period of time after the capture operation, determines the best frame among these images, and then reports the best frame event to the HAL again. Referring to Figure 2, the decision module continues to acquire images in window Y, determines the best frame Ks2 (the second candidate frame) for window Y, and then reports the best frame Ks2 to the HAL.
[0125] At this point, the processing cache module in the HAL can write the RAW format image of Ks2 to the dedicated buffer A for caching non-candidate frames. The processing and capture module can then determine, from Ks2 cached in dedicated buffer A and Ks1 cached in dedicated buffer B, that Ks2, with the higher overall score, is the highlight image. The processing and capture module can then process the highlight image Ks2, outputting it in photo format, overwriting the previously temporarily generated photo and thumbnail of Ks1 as the final photo and thumbnail captured by the user's shooting operation.
[0126] It is understandable that when the comprehensive score of Ks2 is lower than that of Ks1, the processing snapshot module does not need to snapshot again. The photos and thumbnails of Ks1 determined above are the photos and thumbnails taken by the final user.
[0127] Optionally, during the photo output process, the capture processing module may also use other image processing algorithms to optimize the image to obtain a photo with better overall evaluation and enhance the user's shooting experience. Such image processing algorithms include but are not limited to saturation optimization algorithms, exposure compensation algorithms, and facial / body enhancement algorithms.
[0128] In some embodiments, when obtaining the best frame RAW image from the rotating buffer according to the frame number and writing it into the dedicated buffer, the processing cache module can also obtain multiple frames of RAW images including the best frame RAW image from the rotating buffer, and write the fused image obtained after image fusion using these multiple frames of RAW images into the dedicated buffer, further improving the effect and quality of the final photo.
[0129] Exemplarily, the processing cache module can obtain the RAW image Fi of the best frame corresponding to the frame number i, and the RAW images before and after Fi, such as Fi-1 to Fi-3, and Fi+1, from the rotating buffer according to the received frame number i. Optionally, the processing cache module can also only obtain the RAW image Fi corresponding to the frame number i, and the RAW image before Fi. Alternatively, the processing cache module can also only obtain the RAW image Fi corresponding to the frame number i, and the RAW image after Fi, etc. The embodiment of the present application does not impose any specific restrictions on the specific range of multiple frames of RAW images including the target RAW image.
[0130] Taking frames Fi-3 to Fi+1 as an example, the processing and caching module can fuse multiple frames of Fi-3 to Fi+1 to generate a fused image, Fi-N. The processing and caching module can then write the fused image Fi-N to a dedicated buffer. Subsequently, the capture processing module can output a photo and thumbnail based on the fused image.
[0131] Furthermore, the processing buffer module can also determine whether to retrieve one or multiple RAW frames from the rotating buffer based on the shooting mode. For example, in Night mode, the processing buffer module can retrieve multiple RAW frames from the rotating buffer; in non-Night modes, such as Portrait mode or High Dynamic Range mode, the processing buffer module can only retrieve one RAW frame from the rotating buffer.
[0132] Compared with directly inputting the RAW image into the perception module to obtain information such as image content and image quality, the small-sized Tiny image can greatly reduce the computational cost of each detection algorithm in the perception module, increase the rate at which each detection algorithm outputs the corresponding image content information, and effectively avoid frame drops and freezes.
[0133] Furthermore, in one implementation, the electronic device 100 can simultaneously enable all detection algorithms and set the operating frequency of each detection algorithm to be the same as the frequency at which the camera reports images. For example, after receiving each frame of a tiny image, the subject detection algorithm in the perception module can process the frame to obtain the subject in the frame; at the same time, the action evaluation algorithm can process the frame to determine the action score for the frame; and at the same time, the face detection algorithm, face attribute algorithm, and clarity algorithm can all process the frame to obtain the corresponding image content information.
[0134] In another implementation, the electronic device 100 can turn on all detection algorithms at the same time, but the working frequencies of different detection algorithms are not the same. For example, the electronic device 100 can set the working frequency of the subject detection algorithm to be the same as the frequency of the camera reporting images, that is, the subject detection algorithm is executed once per frame. The electronic device 100 can set the working frequency of the motion evaluation algorithm to 1 / 4 of the frequency of the camera reporting images, that is, the motion evaluation algorithm is executed once every 4 frames. The electronic device 100 can set the working frequency of the face detection algorithm to 1 / 2 of the frequency of the camera reporting images, that is, the motion evaluation algorithm is executed once every 2 frames. The electronic device 100 can set the working frequency of the clarity algorithm to be the same as the frequency of the camera reporting images, and so on. The examples are not listed here one by one. In this way, the electronic device 100 can further reduce the computing cost of the perception module and save power consumption.
[0135] In particular, in scenarios where the operating frequencies of different detection algorithms are not the same, the detection algorithm that does not perform processing can output the results obtained from the last detection, thereby avoiding the score being too low due to non-detection, which reduces the stability and accuracy of the wonderful moment detection.
[0136] In another implementation, the electronic device 100 may selectively enable some detection algorithms. Specifically, the electronic device 100 may determine which detection algorithms to enable based on the user-preset objects and / or object motion states. The remaining detection algorithms may be disabled. For example, if the user-set manually captured objects only include cats and dogs, the electronic device 100 may disable the face detection algorithm and the face attribute algorithm.
[0137] 4A-4C are a set of user interfaces for enabling manual snapshot provided in an embodiment of the present application.
[0138] FIG4A is a home page of an electronic device 100 according to an embodiment of the present application. As shown in FIG4A , the home page may include a status bar 111 , a page indicator 112 , and a plurality of application icons.
[0139] The status bar may include one or more signal strength indicators of a mobile communication signal (also referred to as a cellular signal), a wireless fidelity (Wi-Fi) signal strength indicator, a battery status indicator, a time indicator, and the like.
[0140] The page indicator 112 may be used to indicate the positional relationship between the currently displayed page and other pages.
[0141] The multiple application icons may include a settings application icon 113A, an application market application icon 113B, a gallery application icon 113C, a weather application icon 113D, a phone application icon 113E, a browser application icon 113F, a camera application icon 113G, and the like. The main interface may also include other application icons, which are not listed here one by one. The multiple application icons may be distributed across multiple pages. The page indicator 112 may be used to indicate which page the user is currently browsing among the multiple pages hosting multiple applications. The user may use touch operations such as swiping left and right to browse other pages.
[0142] It is understandable that FIG. 4A and the user interfaces described subsequently merely illustrate a possible user interface style of the electronic device 100 taking a mobile phone as an example, and should not constitute a limitation on the embodiments of the present application.
[0143] As shown in FIG4A , electronic device 100 may detect a user operation on camera application icon 113G. In response to the operation, electronic device 100 may launch the camera application and display the user interface of the camera application shown in FIG4B . FIG4B is a user interface for a camera application providing a capture and display service, also known as a preview interface, according to an embodiment of the present application.
[0144] As shown in FIG. 4B , the preview interface may include a mode bar 121 , a shooting control 122 , a preview window 123 , and a review control 124 .
[0145] The mode bar 121 may include multiple shooting mode options, such as "night scene", "portrait", "photo", "video recording", etc. Different shooting modes can provide users with shooting services with different effects. Users can select any shooting mode from multiple shooting modes to shoot according to different needs. For example, "photo recording" may be the default shooting mode for taking photos. "Video recording" is used to record videos. The "night scene" mode is suitable for shooting scenes with dim light, such as at night. The "portrait" mode is suitable for shooting scenes where the subject is a person. The electronic device 100 can also provide more shooting modes, such as "large aperture", "movie", "professional", etc., which are not listed here one by one.
[0146] The electronic device 100 can detect user operations on the mode bar 121 and change the currently used shooting mode according to the user operations. The user operations are, for example, left / right swipe operations. For example, when it is detected that the mode bar 121 is dragged and slid to the left (left swipe operation) and the float 125 stops at the "Portrait" option, the electronic device 100 can switch to the "Portrait" mode. By default, the electronic device 100 first uses the "Photograph" mode.
[0147] The capture control 122 is used to trigger a photo. The electronic device 100 can detect whether a user operation, such as a click, is applied to the capture control 122. Upon detecting a user operation on the capture control 122, the electronic device 100 can generate a capture instruction. Based on the capture instruction, the electronic device 100 can retrieve the image reported by the camera at the corresponding timestamp and save it as a photo. In summary, a user operation on the capture control 122 is the capture operation described in this application.
[0148] The preview window 123 can be used to display the image reported by the camera in real time. The display area corresponding to the preview window 123 can be called the preview area. The area corresponding to the preview window 123 can be called the preview area. In different shooting modes, the electronic device 100 can process the image reported by the camera to improve the display effect of the image. For example, in the "portrait" mode, the electronic device 100 can blur the background of the image reported by the camera to highlight the portrait. Here, the preview window 123 can display in real time the image processed by the image processing algorithm corresponding to the different shooting modes, so that the user can perceive the shooting effects corresponding to the different shooting modes in real time.
[0149] The review control 124 can be used to browse thumbnails of photos / videos that have been taken. The display area corresponding to the review control 124 can be called a thumbnail area. The area corresponding to the review control 124 can be called a thumbnail area. In an embodiment of the present application, when a shooting operation in which the user clicks the shooting control 122 is detected, the electronic device 100 may first display a thumbnail of the best frame determined before (including the moment when the shooting operation occurs) in the review control 124. After obtaining an image with a better comprehensive evaluation within a period of time after the shooting operation, the electronic device 100 may display a thumbnail of the image with a better comprehensive evaluation in the review control 124. Conversely, if the images within a period of time after the shooting operation are not as good as the aforementioned best frame, the thumbnail in the review control 124 may remain unchanged.
[0150] 2 , illustratively, upon detecting a capture operation, electronic device 100 may first display a thumbnail of the best frame Ks1 in review control 124. If the best frame Ks2 in window Y is comprehensively evaluated as superior to the best frame Ks1 before the capture operation, electronic device 100 may update the thumbnail of the best frame Ks1 originally displayed in review control 124 to a thumbnail of the best frame Ks2. Otherwise, the thumbnail of the best frame Ks1 remains displayed in review control 124, indicating that the final captured photo is Ks1.
[0151] The preview interface also includes a quick function area 126. As shown in Figure 4B, quick function area 126 may include controls 126A, 126B, 126C, and 126D. Control 126A can be used to disable the AI photography feature. AI photography automatically identifies the subject and applies different processing to different subjects, resulting in better photos or videos. Control 126B can be used to turn on the flash. Control 126C can be used to select a filter. Control 126D can be used to access additional functions.
[0152] In an embodiment of the present application, the preview interface may further include a control 127. The control 127 may be used to enable manual snapshot. The control 127 may be referred to as a second control.
[0153] As shown in FIG4B , the electronic device 100 can detect a user operation on the control 127. In response to the operation, the electronic device 100 can enable the manual snapshot function. After enabling the automatic snapshot function, the electronic device 100 can, according to the method shown in FIG1 and FIG2 , obtain the most exciting images within a period of time before and after the shooting operation and output them as photos when the shooting operation is detected, thereby assisting the user in shooting and improving the user's shooting experience.
[0154] Correspondingly, after detecting that manual capture is enabled, electronic device 100 may display the user interface (first interface) shown in FIG4C . As shown in FIG4C , electronic device 100 may display control 128 in place of control 127. Control 128 may be used to disable manual capture. At the same time, control 128 may remind the user that manual capture is enabled.
[0155] In some embodiments, manual capture may conflict with other shooting functions. In this case, only one of the conflicting functions can be enabled. For example, manual capture may conflict with AI photography. In this case, the electronic device 100 does not support simultaneous activation of manual capture and AI photography. In the scenario where manual capture is enabled, if a user operation to enable AI photography is detected, the electronic device 100 may enable AI photography and disable manual capture. Conversely, in the scenario where AI photography is enabled, if a user operation to enable manual capture is detected, the electronic device 100 may enable manual capture and disable AI photography.
[0156] As shown in FIG4C , after manual capture is turned on, the electronic device 100 may display a control 151 to replace the original control 126A, indicating that the AI photography function has been turned off.
[0157] Other shooting functions that do not conflict with manual capture can be turned on or off independently. For example, the user can turn on the flash at any time through control 126B and select a filter through control 126C.
[0158] After manual capture is enabled, electronic device 100 may optionally detect a user operation on control 128. In response to this operation, electronic device 100 may disable manual capture. At this point, referring to FIG4B , electronic device 100 may redisplay control 127 in place of control 128, allowing manual capture to be enabled again. Accordingly, electronic device 100 may also redisplay control 126A in place of control 151, thereby re-enabling the AI photography function that was previously disabled due to a conflict.
[0159] Through controls 127 and 128, users can turn manual capture on and off at any time according to their needs. When the user needs manual capture assistance to take more exciting photos, the electronic device 100 can provide the above manual capture service; when the user does not need manual capture, the electronic device 100 can turn off manual capture, saving computing costs, reducing power consumption, and improving the user experience.
[0160] 5A-5F , in the preview interface after manual capture is enabled, the preview window 123 can sequentially display the athlete's motion status captured by the camera at different moments during the long jump.
[0161] As shown in FIG5C , preview window 123 of electronic device 100 displays an image of a person jumping and lying flat, referred to as image K. After viewing image K, the user finds it particularly impressive and clicks capture control 122 to capture a photo similar to image K. Image K is the target image.
[0162] Due to human reaction delay, the electronic device 100 is often unable to detect the user's operation of clicking the shooting control 122 (i.e., the shooting operation) at the moment of displaying image K. As shown in Figure 5D, the electronic device 100 can detect the user's shooting operation when displaying image L (the person is curled up and preparing to land after jumping). Image L is the image after image K. At this time, in the prior art, the electronic device 100 will output image L as the captured photo. However, image L is not the photo the user expected to take, and may even include problems such as jitter and ghosting, which seriously affects the user's shooting experience.
[0163] With reference to the introduction of Figures 1 to 2, in an embodiment of the present application, after manual capture is turned on, the electronic device 100 may first determine the image with the best comprehensive evaluation in each window as the best frame of the window in units of windows with a fixed step size. After detecting the shooting operation, the electronic device 100 may determine a best frame from the first frame image of the current window to the image corresponding to the moment the shooting operation occurs (i.e., the image obtained in the current window). Then, the electronic device 100 may determine the best frame with better comprehensive evaluation from the best frames of different windows that have been determined (e.g., K1 and Ks1) as a candidate frame, i.e., the first candidate frame. Furthermore, the electronic device 100 may generate photos and thumbnails based on the first candidate frame.
[0164] As shown in Figure 5D , upon detecting a capture operation on the capture control 122, the electronic device 100 may first determine the best frame in the two most recent windows prior to the capture operation (including image L at the time of the capture operation): image K (the first image). The electronic device 100 may generate a photo (the first photo) and a thumbnail corresponding to image K. Referring to Figure 5E , the electronic device 100 may first display a thumbnail of image K in the review control 124, prompting the user that a photo has been captured based on the user's capture operation.
[0165] It is understood that the optimal frame determined by the electronic device 100 according to the preset algorithm is not necessarily the target image determined by the user. However, due to the similarity of images over a short period of time, the difference between the optimal frame determined by the electronic device 100 according to the preset algorithm and the target image determined by the user will not be significant. Therefore, the optimal frame determined by the electronic device 100 according to the preset algorithm can be regarded as the target image of the user.
[0166] In some embodiments, the electronic device 100 may also determine the image frame corresponding to the moment when the shooting operation occurs, such as image L shown in Figure 5D, determine the thumbnail corresponding to the image frame, and then display the thumbnail of the image frame in the review control 124.
[0167] The photo and thumbnail of the image K generated above are temporary. At the same time, after detecting the shooting operation, the electronic device 100 may continue to obtain images generated by the camera within a period of time after the shooting operation, such as the window Y shown in Figure 2. The electronic device 100 may determine an optimal frame, i.e., a second candidate frame, from the images generated by the camera within a period of time after the above-mentioned shooting operation. When the second candidate frame is better than the first candidate frame, the electronic device 100 may generate photos and thumbnails based on the second candidate frame, and determine them as the final photos and thumbnails taken. The aforementioned photos and thumbnails generated based on the first candidate frame are overwritten. Conversely, if the second candidate frame is inferior to the first candidate frame, the aforementioned photos and thumbnails generated based on the first candidate frame are the final photos and thumbnails taken.
[0168] As shown in Figure 5E, after detecting the shooting operation, the preview window 123 can display the latest frame of image reported by the camera: image M (second image), and image M is the best frame within a period of time after the shooting operation, that is, the second candidate frame. At this time, the electronic device 100 compares image M and image K. When it is determined that image M is better than image K, that is, the comprehensive score of image M is higher than that of image K, the electronic device 100 can generate a photo (second photo) and a thumbnail based on image M, and overwrite the photo and thumbnail of the aforementioned image K. At this time, the photo finally taken according to the user's shooting operation is image M. As shown in Figure 5F, the electronic device 100 can display a thumbnail of image M in the review control 124, overwriting the original thumbnail of image K, to indicate that the photo taken by the user has been updated.
[0169] 5F , the electronic device 100 may detect a user operation on the review control 124. In response to the operation, the electronic device 100 may display the user interface shown in FIG5G . The electronic device 100 may display the photo (image M) ultimately taken according to the user's shooting operation in area 221.
[0170] On the contrary, if image M is inferior to image K, that is, the comprehensive score of image M is lower than that of image K, the electronic device 100 may determine that the photo obtained based on image M is the final photo taken, and the thumbnail of image M is the thumbnail of the final photo taken. At this time, the electronic device 100 does not need to update the thumbnail displayed in the review control 124. Referring to Figure 5E, the electronic device 100 may keep displaying the thumbnail of image K in the review control 124 until a new thumbnail is generated for the next shot. When a user operation acting on the review control 124 in Figure 5E is detected, in response to the above operation, the electronic device 100 may display the photo (image K) finally taken by the user's shooting operation, refer to Figure 5G, and no further details are given here.
[0171] By implementing the manual snapshot method shown in Figures 5A-5G, electronic device 100 can obtain all images generated by the camera within a period of time before and after the user's shooting operation, and then select the best quality image from these images to output as a photo. The photos captured in this way not only reflect the user's initiative in capturing the photo, but also ensure the quality of the final output photo, which can provide users with a better shooting experience.
[0172] The electronic device 100 is installed with an application that provides a way to browse captured image resources (e.g., photos, videos, etc.), such as a gallery application. Users can use the above application to browse captured image resources. In an embodiment of the present application, after manually capturing a photo, the electronic device 100 may only display the finalized photo in the gallery application.
[0173] Figure 5H illustrates a user interface for previewing captured image resources provided by the gallery application in an embodiment of the present application. As shown in Figure 5H , this interface may only display a preview thumbnail 225 of the final captured image M, without displaying a preview thumbnail of an image K captured temporarily before the capture operation. Figure 5H may be referred to as the third interface.
[0174] In some embodiments, after the thumbnail generated based on the candidate frame before the capture operation (i.e., the first candidate frame) is displayed in the review control 124, the user immediately clicks the review control 124 to view the captured photos. At this time, the electronic device 100 has not yet obtained another candidate frame (i.e., the second candidate frame) based on images within a period of time after the capture operation.
[0175] In the above situation, the electronic device 100 may first display the photo generated based on the first candidate frame, while continuing to acquire the second candidate frame. When the second candidate frame is superior to the first candidate frame, the electronic device 100 may automatically display the photo of the second candidate frame, replacing the photo of the first candidate frame.
[0176] As shown in Figures 5D-5E, after detecting a shooting operation, the electronic device 100 may generate a photo and a thumbnail based on the first candidate frame image K before the shooting operation. After the electronic device 100 displays a thumbnail of image K in the review control 124, referring to Figure 6A, the electronic device 100 may detect a user operation acting on the review control 124. In response to the above operation, the electronic device 100 may display the user interface (second interface) shown in Figure 6B. The electronic device 100 may first display a photo generated based on the first candidate frame image K in area 221 (third area).
[0177] While displaying the interface shown in FIG6B , the electronic device 100 may obtain the best frame within a period of time after the shooting operation, that is, the second candidate frame, such as image M. After confirming that image M is better than the candidate frame image K before the shooting operation, the final photo taken is image M. Referring to FIG6C , the electronic device 100 may automatically display image M in area 221, replacing image K. Conversely, if image M is better than image K, the photo and thumbnail generated based on image K are the final photo and thumbnail. The photo shown in FIG6B is the final photo taken and does not need to be updated.
[0178] After viewing the captured photo: Image M, the electronic device 100 may detect a user operation on the control 224. In response to the operation, the electronic device 100 may redisplay the preview interface, as shown in FIG6D . At this time, the thumbnail displayed in the review control 124 is also updated to the thumbnail of Image M.
[0179] By implementing the above method, the electronic device 100 can give feedback to the user immediately after detecting the shooting operation to avoid affecting the user experience.
[0180] In some embodiments, in the "video recording" mode, the electronic device 100 can implement manual capture.
[0181] As shown in Figure 7A, the electronic device 100 can detect a left swipe operation on the mode bar 121. In response to this operation, the electronic device 100 can switch from the "photo" mode to the "recording" mode, as shown in Figure 7B. As shown in Figure 7B, after entering the "recording" mode, the electronic device 100 can switch the preview interface originally corresponding to the "photo" mode to the preview interface corresponding to the "recording" mode. Specifically, the electronic device 100 can switch the original shooting control 122 of the "photo" mode to the start recording control 311 of the "recording" mode.
[0182] As shown in Figure 7B, the electronic device 100 can detect a user operation on the start recording control 311. In response to the above operation, the electronic device 100 can start recording the video. Correspondingly, the electronic device 100 can display the recording interface shown in Figure 7C. The electronic device 100 can switch the original start recording control 311 to the end recording control 312. At the same time, the electronic device 100 can display a timestamp control 313 in the preview window 123 to indicate the length of recording. As shown in Figure 7C, the recording interface can also include a control 315. The control 315 can be used to receive the user's shooting operation during the recording process and trigger manual capture.
[0183] Figures 7C-7G illustrate a set of video recording interfaces displayed by electronic device 100 during the recording of a long jumper. Preview window 123 sequentially displays the athlete's motion state as captured by the camera at different moments during the long jump. Timestamp control 313 updates as the recording duration increases.
[0184] When starting to record video, the electronic device 100 may determine the image windows according to the method shown in FIG. 1 with a fixed step length, and determine the best frame for each window.
[0185] For example, when the interface shown in FIG7E is displayed, the user may determine that the image frame (image K) displayed in preview window 123 is particularly exciting. In this case, the user may decide to click control 315 to capture image K during the video recording process. Similarly, due to human reaction delay, electronic device 100 may receive the user's capture operation on control 315 at some point after image K. Referring to FIG7F , electronic device 100 may receive the user's capture operation on control 315 when image L is displayed.
[0186] In response to the above-mentioned shooting operation, the electronic device 100 may generate a thumbnail based on the candidate frame before the shooting operation, and then display the thumbnail. Exemplarily, the electronic device 100 may determine that the image K before the above-mentioned shooting operation is a candidate frame. At this time, referring to Figure 7G, the electronic device 100 may display a thumbnail of image K in the review control 124. In a scenario where the images generated by the camera for a period of time after the shooting operation are inferior to image K, the electronic device 100 may determine that the final photo taken is image K. Correspondingly, the thumbnail of image K displayed in the review control 124 will not change.
[0187] Similarly, referring to FIG5F , when the best frame of the image generated by the camera within a period of time after the capture operation (e.g., image M) is better than image K, the electronic device 100 may determine that the final photo captured is image M. Correspondingly, the thumbnail displayed in the review control 124 may also be updated to the thumbnail of image M, which will not be further described here.
[0188] As shown in FIG7G , the electronic device 100 may detect a user operation on the end recording control 312. In response to this operation, the electronic device 100 may end video recording and display the user interface shown in FIG7H . As shown in FIG7H , the electronic device 100 may exit the recording interface and redisplay the preview interface, including redisplaying the start recording control 311 in place of the end recording control 312. Furthermore, the electronic device 100 may display a thumbnail of the newly recorded video in the review control 124.
[0189] As shown in FIG7H , the electronic device 100 may detect a user operation on the playback control 124. In response to the operation, the electronic device 100 may display the user interface shown in FIG7I to display the newly recorded video. As shown in FIG7H , the user may click the play button 314 to view the entire content of the newly recorded video.
[0190] The electronic device 100 may also detect a left swipe operation on the screen. In response to this operation, the electronic device 100 may display the wonderful photos captured by the electronic device 100 during the video recording process using the manual capture strategy. For example, referring to FIG7J , the electronic device 100 may display the manually captured wonderful photo image K. It is understood that the left swipe operation may also be a right swipe operation, which is not limited here.
[0191] In some embodiments, after starting recording, the recording interface may not include the review control 124. After automatically capturing a photo, the electronic device 100 will not display thumbnails of the captured photo in the recording interface. After ending the recording, when browsing the captured images, the electronic device 100 may display the photos automatically captured during the recording process.
[0192] FIG8 is a flowchart of obtaining the best frame of a window provided by an embodiment of the present application.
[0193] S101: Start manual capture.
[0194] Referring to the user interface shown in Figures 4B-4C, electronic device 100 may detect a user operation to enable manual capture. In response to this operation, electronic device 100 may load the program code corresponding to manual capture. After enabling manual capture, electronic device 100 may perform the operations shown in the subsequent steps to obtain the optimal frame of the window.
[0195] S102: Obtain the current frame (frame-b) reported by the camera.
[0196] The camera of electronic device 100 generates a RAW image in real time based on the light within the viewing range and sends the RAW image to the upper-layer application that calls the camera, such as the camera application. For example, at a frame rate of 30fps, the camera reports a RAW image frame to the upper-layer application every 33.3 milliseconds (ms), meaning the upper-layer application receives a RAW image frame every 33.3ms.
[0197] At a certain moment, the upper layer may receive the latest RAW image reported by the camera, such as the i-th RAW image. Referring to Figure 3A, after downsampling and diversion, the electronic device 100 can also obtain the i-th Tiny image at the same time. The electronic device 100 can use small-sized Tiny images for processing and analysis to reduce computing costs and improve processing efficiency. At this moment, the i-th frame image (including the i-th RAW image and the i-th Tiny image) can be called the current frame. The current frame can be recorded as frame-b. The electronic device 100 can use the i-th Tiny image to perform the operations shown in the subsequent steps.
[0198] S103: Determine whether frame-b is the first frame image of the current window.
[0199] The electronic device 100 can determine the current window based on the frame number of frame-b and determine whether frame-b is the first frame image of the window. Exemplarily, the frame number of frame-b is 16. Based on the preset fixed step size of 15 (frames), the electronic device 100 can determine that the current window is window 2 and frame-b is the first frame image of window 2. Exemplarily, the frame number of frame-b is 20. Based on the preset fixed step size of 15 (frames), the electronic device 100 can determine that the current window is window 2 and frame-b is not the first frame image of window 2.
[0200] S104: Write frame-b into the buffer cache.
[0201] The electronic device 100 is provided with a buffer cache. The cache can be used to cache the best frame in the window. When frame-b is the first frame image of the current window, the electronic device 100 can first write frame-b directly into the cache.
[0202] S105: Obtain perception parameters of frame-b.
[0203] If frame-b is not the first frame in the current window, it means that a relatively good frame in the current window is already cached. Electronic device 100 then needs to determine whether frame-b is a better frame. If so, electronic device 100 writes frame-b to the cache, replacing the previously cached frame. This way, after the window ends, the cached image will be the best frame in the window.
[0204] Specifically, the electronic device 100 can determine whether frame-b is a more exciting image through perception parameters. The perception parameters include data reflecting image content and image quality output by the perception module, including interfaces output by various detection algorithms in the perception module and data reflecting image content and image quality reported by the camera and / or other sensors. Referring to the introduction of Figure 3A, the data reflecting image content and image quality output by the perception module include: the object frame output by the subject detection algorithm, the action score output by the action evaluation algorithm, the face frame output by the face detection algorithm, the face performance score output by the face attribute algorithm, the clarity output by the clarity algorithm, the exposure reported by the camera and / or other sensors, etc.
[0205] S106: Determine whether it is a typical scene based on the perception parameters.
[0206] A typical scene refers to a scene in which a preset specific object is captured. The objects in an image of a typical scene include the preset specific object. Specifically, the subject detection algorithm can identify whether the image includes a preset specific object, such as a person, cat, or dog. When such a specific object is identified, the subject detection algorithm can output an object frame. Therefore, electronic device 100 can determine that frame-b is an image of a typical scene by sensing whether the parameters include the object frame.
[0207] When the perception parameters acquired in S105 include an object frame, electronic device 100 may determine that frame-b is an image from a typical scene. Conversely, when the perception parameters acquired in S105 do not include an object frame, electronic device 100 may determine that frame-b is not an image from a typical scene. Specifically, when the perception parameters do not include an object frame, the action score output by the action evaluation algorithm is also null, and the face detection algorithm does not output a face frame; the face expression score is also null.
[0208] S107: Compare the comprehensive scores of frame-b and frame-a cached in the cache; when the comprehensive score of frame-b is higher than that of frame-a, frame-b is better than frame-a.
[0209] 3A , the electronic device 100 may determine a comprehensive score indicating the quality of the comprehensive evaluation of the image content and image quality based on the perception parameters.
[0210] Specifically, when frame-b and frame-a are both images in typical scenes, the electronic device 100 can compare the comprehensive scores of frame-b and frame-a. When the comprehensive score of frame-b is higher than that of frame-a, the electronic device 100 can determine that frame-b is a more exciting image. Then, the electronic device 100 can execute the steps shown in S104 to write frame-b into the cache. At this time, the image cached in the cache is updated to the more exciting frame-b. Conversely, when the comprehensive score of frame-b is lower than that of frame-a, the electronic device 100 will not write frame-b into the cache, and the image cached in the cache will still be frame-a, which is more exciting than frame-b. And so on, when the window ends, the image cached in the cache is the best frame of the window.
[0211] If frame-a is not an image in a typical scene, by default, images in typical scenes are always better than images in atypical scenes. Therefore, the electronic device 100 can directly determine that frame-b is a more exciting image. Subsequently, the electronic device 100 can write frame-b into the cache.
[0212] S108: Determine whether frame-a corresponds to a typical scene.
[0213] When frame-b is not an image from a typical scene, electronic device 100 first needs to determine whether frame-a is an image from a typical scene. If frame-a is an image from a typical scene, typical scenes are, by default, preferred over atypical scenes. Therefore, electronic device 100 can directly determine that frame-a is preferred over frame-b.
[0214] If frame-a is also an image from an atypical scene, the electronic device 100 can compare the comprehensive scores of frame-a and frame-b. However, at this time, all the data indicating the image content of frame-b and frame-a are null values. For example, the subject detection algorithm does not output a valid object frame, and thus there is no object motion score, no valid face frame, and no facial expression score. When the comprehensive score of frame-b is higher than that of frame-a, the electronic device 100 can determine that frame-b is a more exciting image. Then, the electronic device 100 can execute the step shown in S104 and write frame-b to the cache. At this time, the image cached in the cache is updated to the more exciting frame-b. Conversely, when the comprehensive score of frame-b is lower than that of frame-a, the electronic device 100 will not write frame-b to the cache, and the image cached in the cache will still be frame-a, which is more exciting than frame-b.
[0215] Repeat S102-S108. After a window ends, the image retained in the cache is the best frame of the window.
[0216] In the manual snapshot method shown in Figure 2, after detecting the shooting operation, the electronic device 100 obtains the first candidate frame before the shooting operation and the second candidate frame after the shooting operation by default, and then determines the more exciting one from the first candidate frame and the second candidate frame as the final photo taken.
[0217] In other embodiments, the electronic device 100 may distinguish whether the moment when the shooting operation occurs is a typical scene based on the image content.
[0218] In a typical scenario, the electronic device 100 may determine more exciting images before and after the shooting operation and output them as photos according to the method shown in FIG. 2 .
[0219] Conversely, in atypical scenarios, as shown in FIG9 , electronic device 100 may only search for the 3A convergence frame closest to the moment the capture operation occurred. 3A technologies include autofocus (AF), auto exposure (AE), and auto white balance (AWB). An image that converges in AF, AE, and AWB is considered a 3A convergence frame.
[0220] In conjunction with Figure 2, when a shooting operation is detected at time t, the electronic device 100 may first obtain the current frame corresponding to time t, such as the 24th frame image (third image) exemplified in Figure 2, and then determine whether the current frame is an image under a typical scene, that is, whether it includes a preset specific object (first object). When a specific object is not included, the electronic device 100 may obtain multiple frames of images within a period of time before and after including the current frame, such as the 19th to 29th frames. Then, the electronic device 100 may determine the 3A convergence frame closest to the current frame (24th frame image) from the above-mentioned 19th to 29th frames, such as the 20th frame image (fourth image). The electronic device 100 may generate a photo based on the above-mentioned 20th frame image and save it.
[0221] In this way, in typical scenarios, the electronic device 100 can determine the most exciting image near the shooting operation by comparing the previous and subsequent images, thereby ensuring the excitement of the photos taken; in atypical scenarios, the electronic device 100 does not need to care about the image content and can obtain images with higher image quality at a lower cost, thereby ensuring the timeliness of the photos taken, while also reducing computing costs and saving power consumption.
[0222] In some embodiments, the electronic device 100 may also provide an automatic capture function. Automatic capture refers to a shooting method that identifies highlights in real time during the preview process based on a preset algorithm and outputs the highlights of the image frames at those highlights as highlights. Compared to manual capture, automatic capture does not require the user to pay attention to the shooting operation.
[0223] Figures 10A-10E are a set of user interfaces for enabling the automatic snapshot function provided in an embodiment of the present application.
[0224] As shown in Figure 10A, the electronic device 100 may detect a user operation on the control 126D. In response to the above operation, the electronic device 100 may display the user interface shown in Figure 10B.
[0225] This interface can display multiple function option controls, such as "Photo Ratio", "Smart Photo", "Video Resolution", "Video Frame Rate" and other controls, which will not be listed here one by one. The "Photo Ratio" control can be used to set the aspect ratio of the captured photo, such as 4:3, 9:16. The "Smart Photo" control can be used to set special photo taking methods, such as voice-controlled photo taking, gesture-controlled photo taking, and automatic snapshot provided in the embodiments of the present application. The "Video Resolution" control can be used to set the resolution of the video, such as [16:9]1080p. The "Video Frame Rate" control can be used to set the frame rate of the video, such as 30fps.
[0226] As shown in Figure 10B, the electronic device 100 may detect a user operation on the "Smart Photo" control. In response to the above operation, the electronic device 100 may display the user interface shown in Figure 10C.
[0227] This interface can display multiple function option controls under the "Smart Photo" directory, such as "Voice Control Photo," "Gesture Photo," "Smile Capture," and "Automatic Capture." The "Voice Control Photo" control can be used to turn voice control photography on or off, and the "Gesture Photo" control can be used to turn gesture photography on or off. The "Smile Capture" control can be used to turn smile capture on or off. The "Automatic Capture" control can be used to turn the automatic capture provided by the implementation of this application on or off. Among them, voice control photography, gesture photography, smile capture, and traditional shooting operations by clicking the shooting control 122 are non-automatic captures. It can be understood that the manual snapshot in the present application is also a snapshot operation performed in response to the user's operation of clicking the shooting control 122 or voice control operation, but it is different from the traditional photo taking by clicking the shooting control 122. The latter is the traditional photo taking, which is based on the user clicking the shooting control 122 and the corresponding image or corresponding photo obtained according to the time point of the click. When selecting the frame, the image frame closest to the time point of the click will be selected. The manual snapshot in the present application is based on the time point when the user clicks the shooting control 122, and one or more image frames are selected according to preset rules from multiple image frames within a period of time before and / or after the time point.
[0228] As shown in FIG10C , electronic device 100 may detect a user operation on button 141. In response to this operation, electronic device 100 may enable automatic capture. Electronic device 100 may display the user interface shown in FIG10D . As shown in FIG10D , button 141 in FIG10C may be replaced with button 142. Button 142 may be used to disable automatic capture.
[0229] Then, the electronic device 100 may sequentially detect user operations on the control 143 and the control 144. In response to the above operations, the electronic device 100 may redisplay the preview interface, as shown in FIG10E .
[0230] Similarly, if there is a function that conflicts with automatic capture, after turning on automatic capture, the electronic device 100 can automatically turn off the corresponding conflicting function. For example, the electronic device 100 can turn off AI photography after turning on automatic capture. As shown in Figure 10E, the electronic device 100 displays the control 151 to replace the original control 126A to indicate to the user that AI photography is turned off. Accordingly, after turning off automatic capture, the electronic device 100 can automatically turn on the function that was previously turned off due to the conflict. For example, after turning off automatic capture, the electronic device 100 can turn on AI photography again.
[0231] After automatic capture is enabled, the electronic device 100 can identify the image reported by the camera in real time during the preview process and determine whether the image is wonderful. After identifying a wonderful image frame, the electronic device 100 can automatically output the wonderful image frame as a photo. Specifically, the electronic device 100 can evaluate the image content and image quality of each frame based on the perception module and decision module shown in Figure 3A to determine whether the image is wonderful. When the decision module determines that the comprehensive score of the image meets the preset value, the decision module can determine that the frame is a wonderful image frame.
[0232] 11A-11H are a set of user interfaces for implementing automatic snapshots provided in an embodiment of the present application.
[0233] Taking the example of capturing an athlete performing a long jump, as shown in Figures 11A-11E , in the preview interface after automatic capture is enabled, preview window 123 sequentially displays the athlete's motion state captured by the camera at different moments during the long jump. Simultaneously, through the perception module and decision module shown in Figure 3A , electronic device 100 can determine the comprehensive score of each frame during the capture process, thereby determining whether each frame is excellent.
[0234] For example, in a long jump, the flat-flat position after the character takes off can be considered the optimal long jump completion state, which is the state shown in preview window 123 in Figure 11C. In this case, the comprehensive score of the clear image (image K) displayed in preview window 123 in Figure 11C is higher than the preset value. Therefore, electronic device 100 can determine that image K is a wonderful image frame. Subsequently, electronic device 100 can generate the wonderful photo shown in image K and the corresponding thumbnail.
[0235] 11F , after obtaining a wonderful photo and thumbnail, the electronic device 100 may display the thumbnail in the review control 124, thereby prompting the user that a photo has been taken. Furthermore, the electronic device 100 may display a prompt control, such as control 211, near the thumbnail to prompt the user that the photo was automatically captured by the electronic device 100, rather than taken by the user.
[0236] The electronic device 100 can detect user operations on the review control 124. In response to the above operations, the electronic device 100 can display the user interface shown in Figure 11G, showing the wonderful photos corresponding to the thumbnails in the review control 124.
[0237] As shown in FIG11G , the electronic device 100 may display automatically captured wonderful photos (image K) in area 221. The user may then browse the automatically captured wonderful photos through area 221. The user interface may also include prompt controls, such as control 222. When the user browses previously captured photos, the electronic device 100 may use the aforementioned controls to determine that the currently displayed photo is an automatically captured wonderful photo, rather than a photo the user has taken.
[0238] Figure 11H shows a user interface for previewing captured image resources provided by the gallery application according to an embodiment of the present application. In this interface, the electronic device 100 can also display a prompt control, such as control 223, on the preview of automatically captured wonderful photos. In this way, when the user browses all captured photos, the user can use the prompt control to distinguish between automatically captured wonderful photos and photos taken by the user.
[0239] In some scenarios, the user has opened the preview interface and is ready to shoot, but forgets to click the capture control 122, thus missing the wonderful moment. Or, due to a delay in response, the wonderful moment has already ended when the capture control 122 is clicked, and the wonderful photo cannot be captured. By implementing the above method, after turning on automatic capture, the electronic device 100 can automatically identify the wonderful moments in the preview scene and capture the wonderful photos. In this way, the electronic device 100 can assist the user in shooting, so that the user does not miss the wonderful moments.
[0240] In some embodiments, in the "video recording" mode, the electronic device 100 can automatically capture photos.
[0241] In some embodiments, starting from the first frame when switching to the "recording" mode, the electronic device 100 can refer to the operations shown in Figures 11A-11F to capture wonderful moments and capture wonderful photos until switching to other modes that do not support automatic capture or exiting shooting.
[0242] In other embodiments, after switching to "recording" mode and starting recording, the electronic device 100 may refer to the operations shown in Figures 11A-11F to capture wonderful moments and capture wonderful photos until the recording ends. This means that in "recording" mode, the electronic device 100 does not automatically capture photos during the period before recording begins.
[0243] Optionally, the electronic device 100 may enable manual capture and manual capture at the same time. In some embodiments, when the time interval between the wonderful image frame determined by manual capture and the best frame determined by manual capture is very short, the electronic device 100 may save the best frame of manual capture and discard the wonderful image frame of manual capture.
[0244] Specifically, the electronic device 100 may set a time interval t (or frame interval number), for example, 1 second (frame interval number 30 frames). When the time interval between a manually captured wonderful image frame and a manually captured best frame is within 1 second, the electronic device 100 may discard the manually captured wonderful image frame. If the time interval between a manually captured wonderful image frame and a manually captured best frame is greater than 1 second, the electronic device 100 may save the manually captured wonderful image frame and the manually captured best frame.
[0245] In this way, the electronic device 100 can avoid generating photos that are very similar to photos taken by the user, causing redundancy and wasting storage resources.
[0246] FIG12 is a schematic diagram of the system structure of the electronic device 100 provided in an embodiment of the present application.
[0247] The system of electronic device 100 can adopt a layered architecture. As shown in Figure 12, the system layers of electronic device 100 include an application layer, an application framework layer, a hardware abstraction layer, a driver layer, and a hardware layer. The layers communicate with each other through software interfaces.
[0248] The application layer may include a series of application packages. For example, an application package may include a camera application, a gallery application, and the like.
[0249] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a camera access interface. The camera access interface includes camera management and camera devices. The camera access interface is used to provide an application programming interface and programming framework for camera applications.
[0250] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, which provides a virtual hardware platform for the operating system. Exemplarily, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library. Among them, the camera hardware abstraction layer can provide virtual hardware for camera device 1 (first camera), camera device 2 (second camera) and more camera devices. A variety of image processing algorithms are stored in the camera algorithm library. For example, in an embodiment of the present application, the camera algorithm library may include a manual capture algorithm, an automatic capture algorithm, an image processing algorithm, etc. Among them, the manual capture algorithm and the automatic capture algorithm also include the perception algorithms required to implement manual capture / automatic capture, such as the subject detection algorithm, action evaluation algorithm, face detection algorithm, face attribute algorithm, clarity algorithm, etc. shown in Figure 3A.
[0251] The driver layer is the layer between hardware and software. It includes drivers for various hardware components. For example, the driver layer may include a camera device driver, a digital signal processor driver, and an image processor driver. The camera device driver drives the camera's sensor to capture images and the image signal processor to pre-process them. The digital signal processor driver drives the digital signal processor to process images. The image processor driver drives the graphics processor to process images.
[0252] In the implementation of this application, after detecting a user operation to open the camera application (the user operation acting on the camera application icon 113G shown in Figure 4A), the camera application calls the camera access interface of the application framework layer to start the camera application, and then sends an instruction to start the camera application by calling the camera device in the camera hardware abstraction layer, such as camera device 1. The camera hardware abstraction layer sends the instruction to the camera device driver of the driver layer. The camera device driver can start the sensor corresponding to the camera device, such as sensor 1, and then collect the image light signal through the sensor, and transmit the image light signal to the image signal processor for preprocessing to obtain a RAW image, and then transmit the original stream back to the camera hardware abstraction layer through the camera device driver.
[0253] The camera hardware abstraction layer can send the RAW image stream to the camera algorithm library. Based on the support of the digital signal processor and image processor, the camera algorithm library can downsample the RAW image stream to obtain a low-resolution processing stream (Tiny stream).
[0254] Then, based on the support of digital signal processors and image processors, the camera algorithm library can call perception algorithms such as subject detection algorithm, motion evaluation algorithm, face detection algorithm, face attribute algorithm, and clarity algorithm to obtain various data for evaluating image content and image quality.
[0255] In manual capture, based on the aforementioned data and data reported by the camera and / or other sensors, the camera algorithm library determines the best frame for the window and then reports a best frame event to the HAL. In automatic capture, based on the aforementioned data and data reported by the camera and / or other sensors, the camera algorithm library determines whether the image meets preset conditions and, therefore, whether the image is excellent. After identifying an excellent image frame, the camera algorithm library reports an automatic capture event to the HAL.
[0256] In manual capture, upon receiving a best frame event, the HAL retrieves the corresponding frame from the received RAW image stream based on the frame number carried in the event and caches it in a dedicated buffer. Upon detecting a capture operation, the best frame retrieved from the dedicated buffer is output as a stunning photo. During this process, the HAL can also call upon the camera algorithm library to perform additional image processing on the stunning frame that triggered automatic capture, such as saturation optimization, exposure compensation, and facial / body enhancement algorithms, to achieve a superior image.
[0257] In automatic capture, after receiving the automatic capture event, HAL can obtain the corresponding image frame from the received RAW image stream based on the frame number carried in the event, and then output it as a wonderful photo.
[0258] After generating the photo and its thumbnail, the HAL can return the image data to the camera application through the camera access interface. Subsequently, the camera application and the gallery application can display the captured photos to the user with the support of the window manager.
[0259] FIG13 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.
[0260] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 129, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 145, a battery 146, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0261] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0262] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.
[0263] The controller can generate an operation control signal based on the instruction operation code and the timing signal to complete the control of instruction fetching and execution. The processor 110 can also be provided with a memory for storing instructions and data.
[0264] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.
[0265] The I2C interface is a bidirectional synchronous serial bus that includes a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C bus lines. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, and the like via different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K via the I2C interface, enabling communication between the processor 110 and the touch sensor 180K via the I2C bus interface, thereby implementing the touch function of the electronic device 100.
[0266] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the electronic device 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the electronic device 100.
[0267] The USB interface 130 is an interface that complies with USB standards and may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, or the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100, or to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones to play audio. This interface can also be used to connect other electronic devices, such as augmented reality devices.
[0268] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.
[0269] The charging management module 140 is configured to receive charging input from a charger. While the charging management module 140 is charging the battery 146 , it can also power the electronic device through the power management module 145 .
[0270] The power management module 145 is used to connect the battery 146, the charging management module 140, and the processor 110. The power management module 145 receives input from the battery 146 and / or the charging management module 140 to provide power to the processor 110, the internal memory 129, the display 194, the camera 193, and the wireless communication module 160.
[0271] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0272] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.
[0273] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniLED, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device can include one or N display screens 194, where N is a positive integer greater than one.
[0274] In an embodiment of the present application, the electronic device 100 can display the user interface shown in Figures 4A-4C, Figures 5A-5H, Figures 6A-6D, Figures 7A-7J, Figures 10A-10E, and Figures 11A-11H through the display function provided by devices such as the GPU, the display screen 194, and the application processor.
[0275] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.
[0276] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise and brightness. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.
[0277] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
[0278] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.
[0279] In an embodiment of the present application, the electronic device 100 can acquire images and implement manual and / or automatic capture through the shooting functions provided by the ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0280] Internal memory 129 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM). The RAM can be directly read and written by the processor 110 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. The NVM can also store executable programs and user and application data, and can be pre-loaded into the RAM for direct reading and writing by the processor 110.
[0281] In an embodiment of the present application, the computer program code corresponding to manual and / or automatic snapshots can be stored in NVM. When manual and / or automatic snapshots are enabled, the corresponding program code can be loaded into RAM. The processor 110 can directly read the program code in RAM to implement manual and / or automatic snapshots. The wonderful photos captured by manual and / or automatic snapshots and their thumbnail images can be written to NVM for storage and can be read and browsed by the user at any time.
[0282] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, image files such as manually captured and / or automatically captured photos and their thumbnails can be stored in the external non-volatile memory.
[0283] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor. For example, after the camera application is turned on, the camera application can call the audio module 170 to obtain the ambient audio signal and combine it with the image stream generated by the camera and other devices to form a video.
[0284] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A may be located on display screen 194. When a touch operation is performed on display screen 194, electronic device 100 detects the intensity of the touch operation using pressure sensor 180A. Electronic device 100 may also calculate the location of the touch based on the detection signal from pressure sensor 180A.
[0285] Gyroscope sensor 180B can be used to detect the magnitude of the angular velocity of electronic device 100 in various directions (generally three axes). Acceleration sensor 180E can detect the magnitude of the acceleration of electronic device 100 in various directions. Gyroscope sensor 180B and acceleration sensor 180E can be used to determine the motion posture of electronic device 100. In the embodiment of the present application, the data reported by acceleration sensor 180E and gyroscope sensor 180B can be used to determine the motion state of the device, thereby assisting in determining whether the image at the corresponding moment is jittery, has ghosting, etc.
[0286] The air pressure sensor 180C is used to measure air pressure. The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. The distance sensor 180F is used to measure distance. In a shooting scenario, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing. The proximity light sensor 180G can be used to determine whether there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the perceived ambient light brightness. In a shooting scenario, the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The fingerprint sensor 180H is used to collect fingerprints. The temperature sensor 180J is used to detect temperature. The bone conduction sensor 180M can obtain vibration signals.
[0287] The touch sensor 180K is also called a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.
[0288] In an embodiment of the present application, the electronic device 100 can identify whether there is a touch operation acting on the screen and which position and type of touch operation is acting on the screen through the touch detection function provided by the touch sensor 180K, thereby realizing the switching of the user interface and displaying the user interface shown in Figures 4A-4C, 5A-5H, 6A-6D, 7A-7J, 10A-10E, and 11A-11H, so that the user can obtain manual capture / automatic capture services.
[0289] Keys 190 include a power button, volume button, and other buttons. Electronic device 100 can receive key inputs and generate key signal inputs related to user settings and function control of electronic device 100. Motor 191 can generate vibration prompts. Indicator 192 can be an indicator light. SIM card interface 195 is used to connect a SIM card.
[0290] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and Extensible Markup Language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code, such as HTML through <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.
[0291] A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that uses graphics. It can be an icon, window, control, or other interface element displayed on the display of an electronic device. Controls can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.
[0292] As used in the specification and appended claims of the present application, the singular expressions "a", "an", "said", "above", "the" and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items. As used in the above embodiments, the term "when..." can be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...". Similarly, the phrase "when determining..." or "if (stated condition or event) is detected" can be interpreted to mean "if determining..." or "in response to determining..." or "when (stated condition or event) is detected" or "in response to detecting (stated condition or event)", depending on the context.
[0293] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).
[0294] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.
[0295] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.< / canvas> < / video> < / videoview> < / imgview> < / textview>
Claims
1. A shooting method, applied to an electronic device, wherein the electronic device comprises a camera, It is characterized in that The method comprises: Open the Camera app; At time T1, a preview interface is displayed, wherein the preview interface includes a preview area and a thumbnail area, and the preview area displays the image collected by the camera in real time; A shooting operation is detected at time T2; in response to the shooting operation, the thumbnail area displays a thumbnail of a first photo, the first photo is obtained based on a first image, and the first image is an image captured by the camera after time T1 and before time T2; In response to the shooting operation, the thumbnail area displays a thumbnail of a second photo, where the second photo is obtained based on a second image, and the second image is an image captured by the camera after the time T2 but does not include the time T2.
2. The method according to claim 1, It is characterized in that Before the thumbnail area displays the thumbnail of the second photo, the method further includes: detecting a first operation on a thumbnail of the first photo; In response to the first operation, displaying a second interface, the second interface including a third area, the first photo being displayed in the third area; The first photo displayed in the third area is replaced by the second photo.
3. The method according to claim 1 or 2, It is characterized in that The method further comprises: Open the gallery application and display a third interface, wherein the third interface displays one or more images, wherein the one or more images include a preview thumbnail of the second photo but do not include a preview thumbnail of the first photo.
4. The method according to any one of claims 1 to 3, It is characterized in that The preview interface also includes a second control, and the method further includes: detecting a user operation acting on the second control and enabling a manual snapshot function.
5. The method according to any one of claims 1 to 4, It is characterized in that The first image is a frame of image with the highest comprehensive score captured by the camera after the T1 moment and before the T2 moment, and the comprehensive score of the image is determined by one or more of the following: the number, size and / or position of the first object, the completion of the first action, the number, size and / or position of faces, the degree of smile of the faces, the degree of eye opening of the faces, clarity, or exposure; the second image is a frame of image with the highest comprehensive score captured by the camera after the T2 moment and excluding the T2 moment, and the comprehensive score of the second image is higher than the comprehensive score of the first image.
6. The method according to claim 5, It is characterized in that The comprehensive score of the second image is higher than the comprehensive score of the first image, and a difference between the comprehensive score of the second image and the comprehensive score of the first image is higher than a first preset value.
7. The method according to claim 5 or 6, It is characterized in that Before the shooting operation is detected at time T2, the method further includes: selecting a first best frame with the highest comprehensive score from the images collected by the camera from time T3 to time T4; the time T3 and the time T4 are after the time T1 and before the time T2, and the time T3 and the time T4 are after the time T1 and are determined according to a time window with a fixed step length; In response to the shooting operation, the thumbnail area displays a thumbnail of a first photo, where the first photo is obtained according to the first image, specifically including: In response to the shooting operation, the second best frame with the highest comprehensive score is selected from the images captured by the camera from time T4 to time T2; the frame with a higher comprehensive score is determined from the first best frame and the second best frame as the first candidate frame; the first image is the first candidate frame.
8. The method according to claim 7, It is characterized in that In response to the shooting operation, the thumbnail area displays a thumbnail of a second photo, where the second photo is obtained according to the second image, specifically including: Select the second candidate frame with the highest comprehensive score from the images captured by the camera from time T2 to time T5; It is determined that the comprehensive score of the second candidate frame is higher than that of the first candidate frame, and the second image is the second candidate frame.
9. The method according to claim 8, It is characterized in that The time length from the time T2 to the time T5 is the same as or different from the time length from the time T3 to the time T4.
10. The method according to any one of claims 1 to 9, It is characterized in that In response to the shooting operation, before the thumbnail area displays a thumbnail of the first photo, the method further includes: determining that a third image includes a first object, and the third image is an image displayed in the preview area when the electronic device detects the shooting operation.
11. The method according to claim 10, It is characterized in that The method further comprises: When the third image does not include the first object, a thumbnail of a fourth photo is displayed in the thumbnail area, the fourth photo is obtained based on the fourth image, the fourth image is a convergence image closest to the third image, and the convergence image includes satisfying one or more of the following: auto focus convergence, auto exposure convergence, or auto white balance convergence.
12. The method according to any one of claims 1 to 11, It is characterized in that The first photo is obtained according to the first image, specifically including: Acquire multiple frames of images including the first image captured by the camera; The multiple frames of images are fused into a new frame of image, and the first photo is obtained based on the new image.
13. The method according to claim 12, It is characterized in that Before acquiring the preceding and following multiple frames of images including the first image captured by the camera, the method further includes: starting a first shooting mode.
14. The method according to claim 13, It is characterized in that The first shooting mode is a night scene mode.
15. An electronic device, It is characterized in that The method comprises one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program codes, wherein the computer program codes include computer instructions, and when the one or more processors execute the computer instructions, the method according to any one of claims 1 to 14 is executed.
16. A computer-readable storage medium comprising instructions, It is characterized in that When the instructions are executed on an electronic device, the method according to any one of claims 1 to 14 is executed.
