Photographing method and electronic device based on open and close eye detection

Abstract

The application provides a shooting method based on open and close eye detection and an electronic device. After detecting a shooting operation, the electronic device with shooting capability such as a mobile phone or a camera can determine the eye state of an image sequence corresponding to the shooting operation, determine a preferred image and one or more backup images from the image sequence, then display the preferred image and the backup image, and retain one image from the preferred image and the backup image according to a user operation. In this way, the electronic device can improve the shooting experience of the user, so that the user can take photos in accordance with the intention, and the memory of the electronic device is not occupied.

Description

Technical Field

[0001] This application relates to the field of terminals, and in particular to imaging methods and electronic devices based on eye opening and closing detection. Background Technology

[0002] In portrait photography, the subject's blinking can negatively impact the final result. Outdoors, under flash, and other similar conditions, the frequency of blinking is even higher. Consequently, in these environments, users often struggle to capture satisfactory photos. Summary of the Invention

[0003] This application provides an imaging method and electronic device based on eye opening and closing detection.

[0004] Firstly, this application provides a method for taking a picture. This method is applied to electronic devices with shooting capabilities, such as mobile phones and cameras. The method includes: displaying a preview interface showing images captured by a camera; detecting a shooting operation, determining the human eye state of the image sequence corresponding to the shooting operation, and determining a preferred image and one or more substitute images from the image sequence; displaying the preferred image and the substitute images, and retaining one image from the preferred image and the substitute images according to user operation.

[0005] By implementing the method provided in the first aspect, in a portrait shooting scenario, after detecting a user's shooting action, the electronic device can select several frames from a set of images corresponding to the shooting action that are better perceived by the human eye and display them to the user. Furthermore, the user can choose one of these frames that best suits their preferences and save it, thus enhancing the user's shooting experience.

[0006] In some embodiments, displaying a preferred image and a substitute image, and retaining one frame of the preferred image and the substitute image based on user operation, specifically includes: displaying the preferred image; detecting a deletion operation on the preferred image within a first preset time period after displaying the preferred image, deleting the preferred image, and displaying a first substitute image from the substitute images; if no deletion operation on the first substitute image is detected within a first preset time period after displaying the first substitute image, retaining the first substitute image and deleting other substitute images besides the first substitute image. If no deletion operation on the preferred image is detected within the first preset time period after displaying the preferred image, retaining the preferred image and deleting all substitute images.

[0007] By implementing the above method, the electronic device can detect a user deletion operation after displaying the preferred image, and determine whether the user is satisfied with the currently displayed shooting result based on the user's deletion operation. If no deletion operation is detected (i.e., satisfaction), the electronic device can delete other alternative images to reduce redundancy and free up storage space. If a deletion operation is detected (i.e., dissatisfaction), the electronic device can display alternative images for the user to choose from, improving the user's shooting experience.

[0008] In some embodiments, a preferred image and a substitute image are displayed, and one frame of the preferred image and the substitute image is retained according to a user operation. Specifically, this includes: rotating the preferred image and the substitute image; when playing the first image among the preferred image and the substitute image, detecting a user operation on a first control, and in response to the user operation, retaining the first image and deleting the preferred image and / or the substitute image other than the first image.

[0009] By implementing the above method, electronic devices can also determine the user's preferred image through rotating playback and user operation of specific controls, thereby retaining the aforementioned image, deleting other preferred images and / or substitute images, reducing redundancy, and freeing up storage space.

[0010] Preferably, the image sequence corresponding to the shooting operation includes: M1 frame images before the shooting operation and M2 frame images after the shooting operation.

[0011] Preferably, the time range corresponding to the M frames is ≤3s, where M = M1 + M2. In this way, the M frames can cover at least one blink, so the electronic device can select the image of the open eye.

[0012] Preferably, determining one preferred image and one or more substitute images from the image sequence specifically includes: determining the frame with the highest second comprehensive score in the image sequence as the preferred image, and the frame or more with the second highest second comprehensive score as substitute images.

[0013] The second comprehensive score includes the confidence level of the human eye state, and the confidence level is positively correlated with the second comprehensive score. Optionally, the group of human eyes with the highest mean confidence level for both eyes in a frame of an image is considered the target human eye of the image; the confidence level is specifically the mean confidence level of the target human eye. Optionally, the group of human eyes with the highest balanced confidence level for both eyes in a frame of an image is considered the target human eye of the image, where the balanced confidence level for both eyes is the weighted mean of the mean confidence level and the variance of the confidence level for both eyes; the confidence level is specifically the balanced confidence level of the target human eye.

[0014] Optionally, the second comprehensive score also includes sharpness and shutter lag, with sharpness positively correlated with the second comprehensive score and shutter lag negatively correlated with it. Optionally, the second comprehensive score also includes the degree of smile on the target face, with the degree of smile positively correlated with the second comprehensive score.

[0015] In this way, electronic devices can select images with better overall performance in terms of open eyes, faces and / or sharpness, shutter lag, etc., and output them to users as shooting results, thereby improving user satisfaction with the shooting results and enhancing the user's shooting experience.

[0016] In some embodiments, before determining the frame with the highest second comprehensive score in the image sequence as the preferred image, the method further includes: determining whether the current scene is a typical closed-eye scene of portrait photography; determining the frame with the highest second comprehensive score in the image sequence as the preferred image specifically includes: when the scene is an atypical closed-eye scene of portrait photography, determining the frame with the highest second comprehensive score in the image sequence as the preferred image.

[0017] In a typical closed-eye scene, the frame with the highest third comprehensive score in the image sequence is selected as the preferred image, and the frame or more frames with the next highest third comprehensive score are selected as substitute images. The third comprehensive score includes the degree of smile of the target face, but does not include the confidence level of the human eye state. The third comprehensive score is positively correlated with the degree of smile of the target face.

[0018] When shooting in a non-portrait scene, the frame with the highest baseline score in the image sequence is selected as the preferred image, and the frame or more frames with the next highest baseline score are selected as substitute images. The baseline score is the weighted average of sharpness and shutter lag. The baseline score is positively correlated with sharpness and negatively correlated with shutter lag.

[0019] In this way, electronic devices can select different evaluation algorithms in different scenarios, thereby determining the images that users prefer in different scenarios, improving user satisfaction with the preferred images, and enhancing the user's shooting experience.

[0020] In a second aspect, this application provides an electronic device including one or more processors and one or more memories; wherein the one or more memories are coupled to one or more processors, and the one or more memories are used to store a computer program, which, when executed by one or more processors, causes the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0022] Fourthly, this application provides a computer-readable storage medium including a computer program that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0023] Fifthly, this application provides a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0024] Understandably, the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0025] Figure 1 This is a flowchart of a set of imaging methods based on eye opening and closing detection provided in the embodiments of this application;

[0026] Figures 2A-2B This is a schematic diagram of a set of image sequences corresponding to a shooting operation provided in an embodiment of this application;

[0027] Figure 3 This is a schematic diagram illustrating the determination of preferred and alternative images based on confidence levels according to embodiments of this application;

[0028] Figures 4A-4E This is a set of user interface diagrams of a shooting method based on eye opening and closing detection provided in the embodiments of this application;

[0029] Figures 5A-5E This is another set of user interface diagrams provided in the embodiments of this application;

[0030] Figure 6 This is a flowchart of an eye-opening detection method provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of the shutter delay corresponding to the shooting operation provided in the embodiments of this application;

[0032] Figure 8 This is another flowchart of the eye-opening detection provided in the embodiments of this application;

[0033] Figure 9 This is a flowchart of another method for determining preferred and alternative images provided in an embodiment of this application;

[0034] Figure 10 This is a flowchart of another method for determining preferred and alternative images provided in an embodiment of this application;

[0035] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0036] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be a limitation of this application.

[0037] In portrait photography, the subject's blinking can negatively impact the final result. Outdoors, under flash, and other similar conditions, the frequency of blinking is even higher. Consequently, in these environments, users often struggle to capture satisfactory photos.

[0038] Therefore, embodiments of this application provide an imaging method based on eye opening and closing detection.

[0039] This method can be applied to electronic devices with shooting capabilities, such as mobile phones and cameras. Cameras include, but are not limited to, SLR cameras, mirrorless cameras, and action cameras. Beyond mobile phones and cameras, electronic devices implementing the above method can also be tablets, laptops, smart wearable devices (such as smartwatches and smart glasses), in-vehicle devices, smart home devices, and / or smart city devices, etc., all with shooting capabilities. This application does not impose any special limitations on these aspects.

[0040] Figure 1 This is a flowchart of a set of imaging methods based on eye opening and closing detection provided in the embodiments of this application.

[0041] S101. Start the camera to capture images and display the captured images in the photo preview interface.

[0042] The electronic device can detect user actions that activate the camera, such as clicking the camera app icon to open the camera app. In response to such actions, the electronic device can activate the camera to capture an image and display a photo preview interface, showing the image captured by the camera.

[0043] S102. Perform eye-opening detection on the images captured by the camera to determine the state of the human eyes in the images.

[0044] After the camera is turned on, it periodically captures and generates images. Taking a frame rate of 30fps as an example, the camera generates one frame every 33.3ms. Within one second, the electronic device can obtain 30 frames. After each frame is generated, the electronic device can process it using a pre-set eye-opening detection model to determine the human eye state: open / closed. Preferably, the aforementioned eye-opening detection model is a neural network model trained using deep learning.

[0045] S103. Detecting a user's shooting operation, and determining a preferred image and one or more substitute images from the image sequence corresponding to the shooting operation based on the human eye state.

[0046] During the display of the photo preview interface, the electronic device can detect user actions, such as clicking the shutter button. After detecting the user's shooting action, the electronic device can first determine the image sequence corresponding to the shooting action.

[0047] In some embodiments, the electronic device may determine that M frames of images preceding the shooting operation (including the current frame) are the image sequence corresponding to the shooting operation. (See reference) Figure 2A At time Tx, the electronic device can detect the user's shooting operation capture1, which corresponds to the i-th frame image F(i) generated by the camera. At this time, for example, the electronic device can determine that the i-2-th frame image F(i-2), the i-1-th frame image F(i-1), and the current frame F(i) before capture1 are the image sequence corresponding to capture1.

[0048] In other embodiments, the electronic device may determine that M frames (including the current frame) before and after the shooting operation are the image sequence corresponding to the shooting operation. For example... Figure 2B As shown, for example, the electronic device can determine that the (i-1)th frame image F(i-1) before capture1, the current frame F(i), and the (i+1)th frame image F(i+1) after capture1 are the image sequence corresponding to capture1.

[0049] M can be set based on experience. Considering that it takes at least 1 second for a user to complete a blink, it is preferable that the time range corresponding to 1s ≤ M frames ≤ 3s. For example, in a 30fps scene, 30 ≤ M ≤ 90; in a 60fps scene, 60 ≤ M ≤ 180.

[0050] After determining the image sequence corresponding to the shooting operation, based on S102, the electronic device can determine the human eye state of each frame in the image sequence, and then determine a preferred image and one or more substitute images from the image sequence.

[0051] When the electronic device detects that the human eye is open in multiple frames, it can determine the frame with the highest confidence (i.e., the best eye-opening performance) as the preferred image, and the next highest confidence frame or frames as substitute images. Considering the storage space requirements of substitute images and the user's browsing experience when viewing substitute images later, the number of substitute images should not be too large. Preferably, the number of substitute images is 1 ≤ 3.

[0052] Figure 3 This is a schematic diagram illustrating the determination of preferred and alternative images based on confidence levels, provided in an embodiment of this application. Figure 3 As shown, F(1) to F(7) are a set of image sequences corresponding to the shooting operation. Wherein:

[0053] For F(1), the human eye state is closed, and the confidence level is 0; for F(2), the human eye state is closed, and the confidence level is 0.

[0054] For F(3), the eye state is closed, with a confidence level of 0; for F(4), the eye state is open, with a confidence level of 0.33.

[0055] F(5) Eye state = open eyes, confidence level = 0.77; F(6) Eye state = open eyes, confidence level = 0.90;

[0056] F(7) Eye state = open eyes, confidence level = 0.87;

[0057] Based on the confidence level from high to low, the electronic device can obtain: F(6) > F(7) > F(5) > F(4) > F(1), F(2), F(3). Therefore, the electronic device can determine F(6) as the preferred image. In the scenario where two alternative images are set, the electronic device can determine F(7) and F(5) as alternative images.

[0058] S104. In response to the above user shooting operation, output the preferred image and the alternative image, and display a thumbnail of the preferred image on the shooting preview interface.

[0059] Preferably, the electronic device can perform eye-opening detection on the images captured by the camera in a frame-by-frame manner. Taking 3 frames as an example, the electronic device can determine one frame every 3 frames and perform eye-opening detection on that frame. For example, after performing eye-opening detection on the first frame, the electronic device can wait for 2 frames. After receiving the fourth frame, the electronic device then performs eye-opening detection on the fourth frame. Thus, within 1 second, the electronic device can obtain 10 frames of images that have undergone eye-opening detection.

[0060] After Figure 3 After determining the preferred image and the substitute image using the method shown, the electronic device can acquire images near the preferred image and the substitute image, perform multi-frame fusion on the preferred image and the substitute image, further improve the quality of the preferred image and the substitute image, and enhance the user's shooting experience.

[0061] For example, in a frame-skipping scenario, F(5) is the 13th frame image captured by the camera, F(6) is the 16th frame image captured by the camera, and F(7) is the 19th frame image captured by the camera. After determining that F(6) is the preferred image, the electronic device can acquire the 15th and 17th frames near F(6), perform multi-frame fusion on the 15th to 17th frames to obtain the fused preferred image F'(6), and output F'(6) as the user's shooting result. After determining that F(5) and F(7) are the preferred images, the electronic device can acquire the 12th and 14th frames near F(5) and the 18th and 20th frames near F(7), perform multi-frame fusion on the 12th to 14th frames to obtain the fused substitute image F'(5), perform multi-frame fusion on the 18th to 20th frames to obtain the fused substitute image F'(7), and then output the fused substitute images F'(5) and F'(7) as substitute images.

[0062] Understandably, after fusion, electronic devices can perform other processing on the fused preferred and substitute images, such as noise reduction, intelligent brightness adjustment, intelligent color optimization, beautification, and slimming, to further improve the quality of the preferred and substitute images and enhance the user's shooting experience.

[0063] S105. Detect user operation on the thumbnail of the preferred image, display the preferred image and the alternative image, retain one frame of the preferred image and the alternative image according to the user operation, and delete the other images.

[0064] Figures 4A-4E This is a user interface diagram of a shooting method based on eye opening and closing detection provided in the embodiments of this application.

[0065] Upon detecting a user action to activate the camera, the electronic device can start the camera to capture images and display them. Figure 4A The image preview interface shown. Figure 4A As shown, the electronic device can display the image captured by the camera in the preview window 411. After a suitable portrait appears in the preview window 411, the user can control the shooting via the shutter control 412. The operation performed on the shutter control 412 described above constitutes the user's shooting operation.

[0066] After detecting a user's shooting action, the electronic device can determine the image sequence corresponding to the shooting action according to the method described in S103, and then determine a preferred image and a substitute image based on the human eye state of the image sequence. In response to the shooting action, the electronic device can output the preferred image and the substitute image. Figure 4B As shown, the electronic device can display a thumbnail 413 of the preferred image on the camera preview interface. The electronic device can then detect user interaction with the thumbnail 413. In response to the aforementioned user interaction, the electronic device can display... Figure 4CThe image gallery interface is shown. The electronic device can display the complete preferred image 414 in the image gallery interface for the user to browse and view. At this time, the preferred image 414 is the frame with the best eye-opening performance before and after the shooting operation, determined by the electronic device based on eye-opening detection.

[0067] Due to user preferences, the preferred image determined by the electronic device may not be the image the user actually wants. Therefore, after displaying the preferred image 414, the electronic device can detect user interaction with the delete control 421. (Reference) Figure 4D In response to the aforementioned user operation, the electronic device can delete the preferred image 414 and display a replacement image 415 (the first replacement image). Preferably, the electronic device can also display a prompt message 422 to indicate to the user that they are currently browsing a replacement image for the deleted image. When the replacement image 415 also does not meet the user's needs, the user can continue to delete the replacement image 415 using the delete control 421. (See reference) Figure 4E After deleting the replacement image 415, the electronic device can continue to display the next replacement image 416 for the user to browse. After detecting the deletion of the last replacement image, the electronic device then displays other images in the gallery.

[0068] refer to Figure 4C After initially displaying the preferred image 414, the electronic device can observe the subsequent frames for a period of time (a first preset time), such as 10 minutes. If no user operation to delete the preferred image 414 is detected during the observation period, the electronic device can retain the preferred image 414 and delete other alternative images to reduce redundant images and save storage space. Similarly, after initially displaying a certain frame of alternative image (e.g., alternative image 415), the electronic device can also observe the subsequent frames for a period of time (a first preset time), such as 10 minutes. If no user operation to delete the currently displayed alternative image is detected during the observation period, the electronic device can retain the currently displayed alternative image and delete other subsequent alternative images to reduce redundant images and save storage space.

[0069] Figures 5A-5E This is another set of user interface diagrams provided in the embodiments of this application.

[0070] like Figure 5A As shown, in some embodiments, the electronic device may also display control 431 when the preferred image is first displayed. (See reference...) Figures 5B-5D Upon detecting a user interaction with control 431, the electronic device can rotate between displaying the preferred image and the alternative image. Similarly, when displaying the alternative image, the electronic device can display a prompt message 422 to inform the user that they are currently viewing an alternative image. Understandably, while rotating between the preferred and alternative images, the user can also switch the currently displayed image using a left / right swipe touch operation.

[0071] While rotating between preferred and alternative images, the electronic device may display control 432. During the rotation of preferred and alternative images, upon detecting a user interaction with control 432, the electronic device may retain the currently displayed image and delete other preferred and / or alternative images. For example, see reference... Figure 5C The electronic device can detect a user operation on control 432 while displaying the substitute image 416 (the first image). In response to this operation, the electronic device can retain the currently displayed substitute image 416 and delete the preferred image 414 and the substitute image 415. (See reference) Figure 5E After detecting a user operation on control 432, the electronic device can stop displaying controls 431 and 432.

[0072] Similarly, after initially displaying the preferred image 414, the electronic device can observe it for a period of time (a first preset time), such as 10 minutes. If no user operation on control 431 is detected during the observation period, the electronic device can stop displaying control 431 and control 432 and delete other alternative images to reduce redundant images and save storage space.

[0073] Implementation Figures 5A-5E The method shown allows users to more easily browse preferred and alternative images, save images that meet their needs, save user operations, and improve the user's shooting experience.

[0074] Understandably, after starting the camera to capture images and displaying the photo preview interface, the electronic device can also display a feature switch control. After detecting user operation on the aforementioned feature switch control, the electronic device then executes the eye-opening / closing detection-based shooting method shown in S102 to S106.

[0075] Figure 6 This is a flowchart of an eye-opening detection method provided in an embodiment of this application.

[0076] like Figure 6 As shown, the electronic device can be equipped with a zero shutter lag (ZSL) buffer to cache the raw images captured by the camera. The size of the ZSL buffer is greater than or equal to the size of M frames. This allows the electronic device to acquire M frames of images near the user's shooting action at any time.

[0077] Preferably, after receiving the i-th frame of the original image F(i) reported by the camera, the electronic device can first downsample F(i) to obtain a small-sized compressed image f(i). Then, the electronic device inputs the compressed image f(i) into the eye-opening detection model to obtain the human eye information of f(i). The human eye information includes the number of human eyes, the state of the human eyes (open / closed), and the degree of eye opening (reflected by confidence level; the higher the confidence level, the higher the degree of eye opening, i.e., the better the eye-opening performance).

[0078] The electronic device also includes a frame selection module. After detecting a user's shooting action, the frame selection module can refer to... Figure 2A or Figure 2B The method shown determines the M-frame images corresponding to the shooting operation, and then obtains the human eye information of the M-frame images output by the eye-opening detection model. A frame image may include one (left and right eyes) or more sets of human eyes. Preferably, the frame selection module can determine a target human eye from one or more sets of human eyes in a frame image, and then use the human eye information of the target human eye to represent the human eye information of that frame image.

[0079] Optionally, the frame selection module can determine the target eye as the group of eyes with the highest mean confidence score. In scenarios considering a fleeting blink, the frame selection module can also determine the target eye based on the mean confidence score and the variance of the confidence score. The frame selection module can determine the target eye as the group of eyes with the highest equilibrium confidence score. The equilibrium confidence score is the weighted average of the mean confidence score and the variance of the confidence score.

[0080] Then, refer to Figure 3 The method shown allows the frame selection module to compare the mean confidence score (or confidence equilibrium score) of the target human eye in M ​​frames of images, and determine the frame with the highest mean confidence score (or confidence equilibrium score) as the preferred image, and the frame or more frames with the next highest mean confidence score (or confidence equilibrium score) as the substitute images.

[0081] In some embodiments, when multiple sets of human eyes are included, the frame selection module can also calculate the average confidence score of these multiple sets of human eyes, and use the average confidence score of the multiple sets of human eyes (also called the total confidence score) to evaluate the frame image. Thus, the frame selection module can compare the total confidence score of M frames and determine the frame with the highest total confidence score as the preferred image, and the frame or more frames with the next highest total confidence score as the substitute images.

[0082] like Figure 6 As shown, preferably, after determining the M frames corresponding to the shooting operation, the frame selection module can also obtain the sharpness and shutter lag (SL) of these M frames. SL indicates the distance between the image and the shooting operation. For example, refer to... Figure 7Assume the time it takes for the electronic device to detect the capture operation (capture1) is Tx, corresponding to F(i). In a scenario with a frame rate of 30fps (i.e., a frame interval of 33.3ms), the SL of F(i) is 0, the SL of F(i-1) is 0.0333, the SL of F(i-2) is 0.0666, the SL of F(i-3) is 0.0999, and the SL of F(i+1) is 0.0333. For example... Figure 7 As shown, a larger SL indicates a greater distance between the image and the shooting operation; conversely, a smaller SL indicates a closer distance between the image and the shooting operation.

[0083] Then, the frame selection module can determine the preferred image and the substitute image based on the mean confidence score of the target human eye (or the balanced confidence score of the target human eye, or the total mean confidence score of the M-frame images) and the sharpness and SL.

[0084] For example, the frame selection module can determine the overall score W1 for each frame of the image.

[0085] W1=a*average+b*sp+c*(1-SL);

[0086] Where a, b, and c are weights, average is the confidence mean (which can be the confidence mean of the target human eye, the confidence equilibrium value of the target human eye, or the total confidence mean of the image), and sp is the sharpness.

[0087] At this point, after determining W1 for each frame in the M-frame image, the frame selection module can determine the frame with the highest W1 as the preferred image, and the next highest W1 as the substitute image.

[0088] Figure 8 This is another flowchart for eye-opening detection provided in an embodiment of this application.

[0089] like Figure 8 As shown, the electronic device can also be equipped with a face detection model. After downsampling a frame of image (e.g., F(i)) reported by the camera, the electronic device can first input the downsampled compressed image (e.g., f(i)) into the face detection model to obtain the face image in the compressed image. After obtaining at least one face image, the electronic device then inputs the compressed image into the eye-opening detection model to obtain the eye state of the at least one face image. If no face is recognized, the electronic device will not input the current compressed image into the eye-opening detection model. In this way, the electronic device can reduce the computational load of the eye-opening detection model and reduce power consumption.

[0090] During the process of acquiring face images in the compressed image using a face detection model, the face detection model can also output face information for each face image, including but not limited to the area ratio of the face image in the entire image, the degree of frontal view, and the degree of smile. At this point, the frame selection module can comprehensively evaluate each frame in the M-frame image by combining face information, eye information and / or sharpness, and SL, thereby determining the preferred image and the substitute image.

[0091] Similarly, an image frame may include one or more faces. Preferably, the frame selection module can determine a target face from one or more faces in an image frame, and then use the face information of the target face to represent the face information of the image frame. For example, the frame selection module can determine the face with the largest area in an image frame as the target face. In this case, the eyes of the target face are the target eyes.

[0092] Therefore, based on the target face and target eyes, the frame selection module can determine the comprehensive score W2 (second comprehensive score) for each frame image.

[0093] W2=a*average+b*sp+c*(1-SL)+d*ar+e*o-face+f*s-face;

[0094] Where a, b, c, d, e, and f are weights, average is the average confidence level (which can be the average confidence level of the target human eye or the equilibrium confidence level of the target human eye), sp is the sharpness, ar is the area ratio of the target human face, o-face is the frontal orientation of the target human face, and s-face is the smiling orientation of the target human face.

[0095] At this point, after determining W2 for each frame in the M-frame image, the frame selection module can determine the frame with the highest W2 as the preferred image, and the next highest W2 as the substitute image.

[0096] In some embodiments, when multiple faces are included, the frame selection module may also calculate the average area ratio, average frontal face degree, and average smiling face degree of these multiple faces, and use the above average values ​​to evaluate the frame image. In this case, average is the average total confidence score of the image, o-face is the average frontal face degree of one or more faces, and s-face is the average smiling face degree of one or more faces.

[0097] The frame selection module may also employ other strategies to determine the preferred and alternative images. This application does not limit this approach.

[0098] In typical portrait photography scenarios, users prefer photos of people with their eyes open. However, in certain special scenarios, such as birthday wishes, meditation, non-sleep / feigned sleep, or makeup / beauty treatments, users prefer photos of people with their eyes closed. These special scenarios are also known as typical closed-eye scenarios.

[0099] In view of this, embodiments of this application provide another method for determining preferred and alternative images. Figure 9 This is a flowchart of another method for determining preferred and alternative images provided in an embodiment of this application.

[0100] Electronic devices can be pre-configured with special scene recognition models. After a shooting operation is detected, the frame selection module can first input the M-frame images corresponding to the shooting operation into the special scene recognition model to determine the current shooting scene.

[0101] like Figure 9 As shown, when a portrait shooting scene is detected and it falls within a preset special scene (i.e., a typical closed-eye scene), the frame selection module can determine the comprehensive score W3 of each frame image, identifying the frame with the highest W3 as the preferred image, and one or more frames with the next highest W3 as substitute images; conversely, when a portrait shooting scene is detected but not within a special scene (i.e., an atypical closed-eye scene, i.e., an open-eye scene), the frame selection module can proceed according to... Figure 8 The method shown (W2) determines the frame with the highest W2 score as the preferred image, and the next highest W2 frame or more as substitute images. When a non-human portrait shooting scene is detected, the frame selection module can determine the frame with the highest base score as the preferred image, and the next highest base score frame or more as substitute images. The base score is the weighted average of sharpness and (1-SL).

[0102] Specifically, when a portrait shooting scene is identified and it falls within a preset special scene, the frame selection module can determine the comprehensive score W3 (third comprehensive evaluation) for each frame image.

[0103] W3=b*sp+c*(1-SL)+d*ar+e*o-face+f*s-face;

[0104] Therefore, the frame selection module can determine the frame with the highest W3 as the preferred image, and the next highest W3 frame or more as the substitute image.

[0105] Here, o-face represents the frontal orientation of the target face, or the average frontal orientation of one or more faces; s-face represents the smiling orientation of the target face, or the average smiling orientation of one or more faces.

[0106] Figure 10 This is a flowchart of another method for determining preferred and alternative images provided in an embodiment of this application.

[0107] like Figure 10As shown, in some embodiments, when a preset special scene is detected, the frame selection module can sort the M frames in the ZSL buffer from largest to smallest based on the number of faces in the images. Based on this sorting result, the frame selection module can further sort the M frames from largest to smallest based on the number of frontal faces (faces with a frontal appearance greater than a threshold are considered frontal faces, otherwise they are considered side faces). Continuing, based on the above sorting result, the frame selection module can further sort the M frames from largest to smallest based on the number of smiling faces (faces with a smiling appearance greater than a threshold are considered smiling faces, otherwise they are considered non-smiling faces). Finally, based on the above sorting result, the frame selection module can further sort the M frames from largest to smallest based on the image's base score. After the final sorting, the frame selection module can determine the image ranked first as the preferred image, and the one or more frames following the preferred image as substitute images.

[0108] When no special scene is detected, the frame selection module sorts the M frames in the ZSL buffer from largest to smallest based on the number of faces in the images. Based on this sorting, the module further sorts the M frames from largest to smallest based on the number of frontal faces. Continuing, based on this sorting, the module further sorts the M frames from largest to smallest based on the number of open eyes (faces with eyes open above a threshold are considered open, otherwise closed). Finally, based on this sorting, the module further sorts the M frames from largest to smallest based on the image's baseline score. After this final sorting, the module determines the image ranked first as the preferred image, and the one or more frames following the preferred image as substitute images.

[0109] Understandably, in some implementations, electronic devices may not require additional special scene recognition models. The frame selection model can directly determine whether the current scene is a portrait shooting scene or a preset special scene based on the face information output by the face detection model and the eye information output by the eye-opening detection model.

[0110] For example, when the number of faces in each frame of the ZSL buffer is 0, or when the number of faces in more than a preset number of images is 0, the frame selection module can determine that the current scene is not a portrait shooting scene. After determining that the current scene is a portrait shooting scene based on the number of faces, the frame selection module can determine whether the current scene is a preset special scene based on the number of open eyes. When the number of open eyes in each frame of the ZSL buffer is 0, or when the number of open eyes in more than a preset number of images is 0, the frame selection module can determine that the current scene is a special scene; otherwise, the frame selection module can determine that the current scene is not a special scene.

[0111] In this way, electronic devices can reduce the use of network models, save computing resources, and reduce the power consumption of the method.

[0112] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0113] like Figure 11 As shown, the electronic device may include a processor 81, an internal memory 82, an external memory interface 83, a display screen 84, a camera 85, a communication module 86, an audio module 87, a sensor module 88, etc.

[0114] The electronic device may include one or more processors 81. Processor 81 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0115] The processor 81 is connected to one or more internal memories 82. These internal memories 82 include random access memory (RAM) and non-volatile memory (NVM). RAM can be directly read and written by the processor 81 and can be used to store executable programs (e.g., machine instructions) of the operating system or other running programs, as well as user and application data. NVM can also store executable programs and user and application data. The executable programs and data stored in NVM can be pre-loaded into RAM for direct reading and writing by the processor 81. The processor 81 may also include a cache unit, which can be used to store recently used or repeatedly used instructions or data.

[0116] In this embodiment, the computer program code for the shooting method based on eye-opening / closing detection can be stored in the NVM. After detecting a user operation to activate the camera, the electronic device can load the computer program code for the shooting method stored in the NVM into the processor 81 for execution, thereby achieving... Figure 1 The shooting method shown is based on eye opening and closing detection, which outputs portrait photos with open eyes, improving the user's shooting experience.

[0117] Optionally, some electronic devices may also be equipped with an external memory interface 83. The external memory interface 83 can be used to connect to an external NVM to expand the storage capacity of the electronic device. The computer program code of the above-described eye-opening detection-based imaging method can also be stored in an external NVM connected via the external memory interface 83.

[0118] Display screen 84 is used for display. Display screen 84 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be manufactured using organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), miniled, microled, micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. Electronic devices may include one or more display screens 84.

[0119] Camera 85 is used to capture images. An object is projected onto a photosensitive element by an optical image generated through a lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP (Internet Service Provider) for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP (Digital Signal Processor) for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. The electronic device may include one or more cameras 85.

[0120] Electronic devices can utilize the shooting and display capabilities provided by ISP, camera 85, video codec, GPU, display 84, and application processor to perform... Figure 1 The shooting method shown demonstrates Figures 4A-4E or Figures 5A-5E The user interface shown.

[0121] The audio module 87 includes a microphone 87A and a speaker 87B. The microphone 87A, also called a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. The speaker 87B, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. In this embodiment, after the camera is activated, the electronic device can also turn on the microphone 87A to collect sound signals.

[0122] Optionally, the electronic device may include a communication module 86. The communication module 86 includes a wireless communication module and a mobile communication module. The wireless communication module can be used to provide wireless communication solutions for applications on the electronic device, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth Low Energy (BLE), Global Navigation Satellite System (GNSS), frequency modulation (FM), near-field communication (NFC), and infrared (IR) technologies. The mobile communication module is used to provide mobile communication solutions for applications on the electronic device, including 2G / 3G / 4G / 5G technologies. The electronic device can achieve wireless communication functions through the communication module 86, such as sharing captured portraits with other electronic devices.

[0123] Sensor module 88 includes touch sensor 88A. Touch sensor 88A, also known as a "touch device," can be located on display screen 84. The touch sensor 88A and display screen 84 together form a touchscreen, also known as a "touchscreen." Touch sensor 88A is used to detect touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event, such as... Figures 4A-4E , Figures 5A-5E The click operation is shown. Based on the detected touch operation, the electronic device can provide visual output related to the touch operation through the display screen 84.

[0124] Not limited to the touch sensor 88A, the sensor module 88 may also include other sensors, such as pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, ambient light sensors, bone conduction sensors, etc., to enable electronic devices to achieve richer sensing capabilities.

[0125] The processor 81, internal memory 82, external memory interface 83, display screen 84, camera 85, communication module 86, audio module 87, and sensor module 88 are connected via a bus and communicate and exchange data based on the aforementioned bus. The aforementioned bus includes, but is not limited to, an inter-integrated circuit (I2C) bus, an inter-integrated circuit sound (I2S) bus, a pulse code modulation (PCM) bus, a universal asynchronous receiver / transmitter (UART) bus, a mobile industry processor interface (MIPI) bus, a general-purpose input / output (GPIO) interface bus, and / or a universal serial bus (USB) interface bus.

[0126] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device. Optionally, the electronic device may also include more components, such as buttons, motors, indicators, subscriber identification module (SIM) card interfaces, etc. The embodiments of this application do not impose such limitations.

[0127] The term "user interface (UI)" used in the specification, claims, and drawings of this application refers to the medium through which an application or operating system interacts and exchanges information with the user. It converts information from its internal form to a form acceptable to the user. The user interface of an application is written in source code using specific computer languages ​​such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the terminal device, ultimately presenting user-recognizable content such as images, text, and buttons. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, images, and text. The attributes and content of controls in the interface are defined using tags or nodes, such as XML tags. <textview> 、 <imgview> 、

[0128] <videoview>Nodes define the controls contained in the interface. A node corresponds to a control or property in the interface, and after parsing and rendering, the node is presented as the content visible to the user. In addition, many applications, such as hybrid applications, often contain web pages within their interfaces. A web page, also known as a webpage, can be understood as a special control embedded in the application interface. Web pages are source code written in a specific computer language, such as Hypertext Markup Language (HTML), Cascading Style Sheets (CSS), JavaScript (JS), etc. Web page source code can be loaded and displayed as user-readable content by a browser or a web page display component with browser-like functionality. The specific content contained in a webpage is also defined through tags or nodes in the webpage source code; for example, HTML uses tags or nodes to define the content. 、 、 <video> 、 <canvas>Used to define the elements and attributes of a webpage.

[0129] The most common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be an icon, window, control, or other interface element displayed on the screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0130] As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the listed items. As used in the above embodiments, depending on the context, the term "when" can be interpreted as meaning "if..." or "after..." or "in response to determining..." or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining..." or "in response to determining..." or "when (the stated condition or event) is detected" or "in response to detecting (the stated condition or event)."

[0131] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A shooting method applied to electronic devices, characterized in that, The method includes: A preview interface is displayed, showing images captured by the camera; Upon detecting a shooting operation, the human eye state of the image sequence corresponding to the shooting operation is determined, and a preferred image and one or more alternative images are determined from the image sequence. Display the preferred image and the alternative image, and retain one frame of the preferred image and the alternative image according to user operation.

2. The method of claim 1, wherein, The process of displaying the preferred image and the alternative image, and retaining one frame from the preferred image and the alternative image based on user operation, specifically includes: Display the preferred image; If a deletion operation on the preferred image is detected within a first preset time period of displaying the preferred image, the preferred image is deleted, and the first alternative image in the alternative images is displayed; If no deletion operation on the first substitute image is detected within a first preset time after the first substitute image is displayed, the first substitute image is retained, and other substitute images other than the first substitute image are deleted.

3. The method of claim 2, wherein, The method further includes: If no deletion operation is detected on the preferred image within a first preset time period of displaying the preferred image, the preferred image is retained, and all substitute images are deleted.

4. The method of claim 1, wherein, The process of displaying the preferred image and the alternative image, and retaining one frame from the preferred image and the alternative image based on user operation, specifically includes: The preferred image and the alternative image are displayed in a rotating manner. When playing the first image among the preferred image and the alternative image, a user operation on the first control is detected. In response to the user operation, the first image is retained, and the preferred image and / or alternative image other than the first image are deleted.

5. The method of claim 1, wherein, The image sequence corresponding to the shooting operation includes: M1 frame images before the shooting operation and M2 frame images after the shooting operation.

6. The method according to claim 5, characterized in that, The time range corresponding to the M frames is ≤3s, where M=M1+M2.

7. The method according to claim 1, characterized in that, The step of determining a preferred image and one or more substitute images from the image sequence specifically includes: determining the frame with the highest second comprehensive score in the image sequence as the preferred image, and the frame or more with the second highest second comprehensive score as substitute images.

8. The method according to claim 7, characterized in that, The second comprehensive score includes a confidence level of the human eye state, which is positively correlated with the second comprehensive score.

9. The method according to claim 8, characterized in that, The pair of eyes with the highest mean confidence scores for the left and right eyes in a frame of an image is the target eye of the image; the confidence score specifically refers to the mean confidence score of the target eye.

10. The method according to claim 8, characterized in that, The set of human eyes with the highest left and right eye confidence balance values ​​in a frame of an image is the target human eye of the image. The left and right eye confidence balance value is the weighted average of the left and right eye confidence mean and confidence variance. The confidence value is specifically the confidence balance value of the target human eye.

11. The method according to claim 7, characterized in that, The second comprehensive score includes sharpness and shutter lag, wherein sharpness is positively correlated with the second comprehensive score and shutter lag is negatively correlated with the second comprehensive score.

12. The method according to claim 7, characterized in that, The second comprehensive score includes the degree of smile on the target face, and the degree of smile is positively correlated with the second comprehensive score.

13. The method according to claim 7, characterized in that, Before determining the frame with the highest second comprehensive score in the image sequence as the preferred image, the method further includes: determining whether the current scene is a typical closed-eye scene in portrait photography; The step of determining the frame with the highest second comprehensive score in the image sequence as the preferred image specifically includes: in an atypical scene of portrait photography with eyes closed, determining the frame with the highest second comprehensive score in the image sequence as the preferred image.

14. The method according to claim 13, characterized in that, The method further includes: In the typical closed-eye scenario, the frame with the highest third comprehensive score in the image sequence is determined as the preferred image, and the frame or more frames with the next highest third comprehensive score are the substitute images; the third comprehensive score includes the degree of smile of the target face, but does not include the confidence level of the human eye state, and the third comprehensive score is positively correlated with the degree of smile of the target face.

15. The method according to claim 13, characterized in that, The method further includes: When shooting in a non-portrait scene, the frame with the highest basic score in the image sequence is determined as the preferred image, and the frame or more frames with the next highest basic score are designated as substitute images. The basic score is a weighted average of sharpness and shutter lag, and the basic score is positively correlated with the sharpness and negatively correlated with the shutter lag.

16. An electronic device, characterized in that, It includes one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store a computer program that, when the one or more processors execute the computer program, causes the method as described in any one of claims 1-15 to be performed.

17. A chip system applied to an electronic device, the chip system comprising one or more processors, characterized in that, The processor is used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-15.

18. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-15.

19. A computer-readable storage medium comprising a computer program, characterized in that, When the computer program is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-15.

Images