Image recommendation method and related device

Abstract

The present application discloses an image recommendation method and related devices. The electronic device can determine the theme, score and repetitiveness of multiple images, and then recommend images to the user from the multiple images based on the above factors, so that the user can pay attention to the recommended images.

Description

Technical Field

[0001] The present application relates to the field of terminals and image analysis, and in particular to image recommendation methods and related devices. Background Art

[0002] Smart devices such as mobile phones are equipped with camera functions, and users can use the camera function to take the images they want, such as photos, dynamic photos, videos, etc. After users take a large number of photos and videos, it is difficult to accurately find the photos or videos they want. Therefore, some smart devices provide solutions for recommending images, which can recommend some images for users to view, which can save users time and is convenient and fast. In order to improve user experience, how smart devices recommend images is the direction of current and future research. Summary of the invention

[0003] The present application provides an image recommendation method and related devices, which can recommend images to users.

[0004] In a first aspect, an image recommendation method is provided, which is applied to an electronic device. The method may include: displaying a preview interface, wherein the preview interface displays a preview image frame reported by a camera of the electronic device; detecting a user operation for shooting an image, saving a first image, wherein the first image includes one or more preview image frames displayed on the preview interface; determining the theme, score, and repeatability of the first image, wherein the score of the first image is obtained according to any one or more of the image quality, composition, facial beauty, and character action of the first image, and the repeatability indicates whether the first image is repeated with a historically shot image; determining a recommended image from a plurality of first images according to their respective scores, themes, and repeatability; and displaying indication information of the recommended image.

[0005] By implementing the method of the first aspect, the electronic device can recommend images to the user based on the subject, score, and repetitiveness of each image among multiple captured images.

[0006] In combination with the first aspect, in some embodiments, the method may further include: during display of the preview interface, determining a score of each preview image frame displayed on the preview interface, wherein the score of the preview image frame is obtained according to any one or more of image quality, composition, facial beauty, and character action of the preview image frame. Determining the score of the first image specifically includes: determining the score of the first image according to the scores of one or more preview image frames contained in the first image.

[0007] Based on the above implementation, the score of the preview image frame is determined in real time when the image is previewed, which can save the execution time of the subsequent determination of the first image score and reduce the time cost. In addition, since the storage capacity of the preview stream is small, this also facilitates the convenience of the calculation and analysis process.

[0008] In combination with the previous embodiment, the score of each preview image frame displayed on the preview interface can be determined in the following manner: a first score is determined according to the image quality of the jth preview image frame, where the jth preview image frame is any preview image frame displayed on the preview interface; whether the jth preview image frame contains a subject is determined, and if so, a second score of the jth preview image frame is determined according to the composition; if not, the second score, the third score, and the fourth score of the jth preview image frame are all set to 0; when the jth preview image frame contains a subject, the third score of the jth preview image frame is determined according to the subject category of the jth preview image frame; when the subject contained in the jth preview image frame is a person, the fourth score of the jth preview image frame is determined according to the action of the person; a weighted average score of the first score, the second score, the third score, and the fourth score is calculated; and the weighted average score is determined as the score of the jth preview image frame.

[0009] In this way, the score of each preview image frame can reflect the image quality, composition, facial beauty and character action of the preview image frame, so that the score of the final first image can also reflect the above factors.

[0010] Optionally, the first score, the second score, the third score, and the fourth score are all within a first score interval, wherein the first score interval depends on a preset scoring system, such as a ten-point system, a hundred-point system, and the like.

[0011] Optionally, the electronic device may determine the first score according to the clarity and / or symmetry of the jth preview image frame; the higher the clarity and / or symmetry, the higher the first score.

[0012] Optionally, the electronic device may determine the third score of the j-th preview image frame according to the following method:

[0013] In the case where the subject of the j-th preview image frame includes an animal, if the j-th preview image frame includes the face of the animal, the third score is set to a high score, where the high score includes a score whose difference from the full score is less than a preset value, and the full score is the highest score in the scoring range of the third score;

[0014] or,

[0015] When the subject of the jth preview image frame includes a person, the third score is determined based on any one or more of whether the jth preview image frame contains a face, whether the eyes of the contained face are open or closed, the expression of the face, the posture of the face, the area of ​​the face in the jth preview image frame, the size of the area, the size of the proportion, and the number of faces.

[0016] For animals, as long as the face is captured, the image content is acceptable to the user, because the facial beauty can be given a high score. However, users have higher requirements for the beauty of human faces, so the facial beauty of the task can be scored by considering factors such as the facial expression, facial posture, the proportion of the face in the image, and the number of faces. This scoring method is more in line with user expectations.

[0017] Optionally, when the subject of the j-th preview image frame includes a person, the electronic device may determine the third score by the following formula:

[0018]

[0019] for the third score;

[0020] N face is the number of faces contained in the j-th preview image frame;

[0021] s e,i is the score of the i-th face’s eyes opening and closing. When the i-th face opens its eyes, s e,i Set to the first value, when the i-th face closes its eyes, s e,i is set to a second value, the first value being higher than the second value;

[0022] s m,i is the smile score of the i-th face. When the i-th face smiles, s m,i Set to the third value, when the i-th face is not smiling, s m,i The fourth value is set, and the third value is higher than the fourth value;

[0023] s d,i is the posture score of the i-th face. When the i-th face faces the camera, s d,i Set to the fifth value, when the i-th face is not facing the camera, s d,i Set to the sixth value, the fifth value is higher than the sixth value;

[0024] s b is the influence coefficient of the face size of the i-th face. When the size of the i-th face is within the first range, s b Set to the seventh value, when the size of the i-th face is outside the first range, s b Set to the eighth value, the seventh value is greater than the eighth value;

[0025] s N is the face ratio coefficient of the i-th face. The larger the area ratio of the i-th face, the higher the s N The bigger;

[0026] s cis the composition beauty coefficient. When the i-th face is located in the first area of ​​the j-th preview image frame, s c is set to the ninth value, when the i-th face is outside the first area of ​​the j-th preview image frame, s c is set to the tenth value, the ninth value is higher than the tenth value, and the first area includes the central area of ​​the jth preview image frame;

[0027] s p is the face-free penalty threshold. When the jth preview image frame contains a face, s p When the jth preview image frame does not contain a face, s p Set to the preset faceless penalty threshold.

[0028] The above formula provides a specific way to determine a score for measuring the beauty of a person's face.

[0029] Optionally, in the above formula, the first value, the third value, and the fifth value are all high scores; the second value, the fourth value, and the sixth value are all low scores; among them, the high score includes the score whose difference with the full score is less than the preset value, the low score includes the score whose difference with the full score is greater than the preset value, and the full score is the highest score in the scoring range of the third score.

[0030] Optionally, in the above formula, when the distance between the i-th face and the electronic device is different, the same i-th face size corresponds to different s b In this way, the beauty of the face can be evaluated comprehensively at different distances, limiting the scores of faces that are too large or too small.

[0031] Optionally, the electronic device may determine the fourth score of the jth preview image frame in the following manner: determine whether the character in the jth preview image frame includes a preset wonderful action; if so, the fourth score is equal to the product of the confidence level output by the action evaluation output algorithm and the high score; if not, the fourth score is set to a low score; wherein the high score includes a score whose difference from the full score is less than a preset value, the low score includes a score whose difference from the full score is greater than a preset value, and the full score is the highest score in the scoring range of the fourth score.

[0032] In combination with the previous embodiment, in some embodiments, the score of the first image is determined based on the scores of preview image frames contained in the first image, specifically including: determining the score of the first image based on the first preview image frame, the first preview image frame including preview image frames among all preview image frames of the first image except transition frames, the transition frames including preview image frames whose information amount and image quality do not meet the first threshold.

[0033] By removing the transition frame, the score of the first image can more objectively reflect the quality of the first image.

[0034] In combination with the previous embodiment, the electronic device may determine the score of the first image based on the first preview image frame in the following manner: determine the wonderful frame with the highest score in the first preview image frame; determine the weighted average score of the wonderful frame, a first preset number of preview image frames before the wonderful frame in the first preview image frame, and a second preset number of preview image frames after the wonderful frame in the first preview image frame; and determine the weighted average score as the score of the first image.

[0035] Optionally, the first preset number and the second preset number may be the same or different.

[0036] Optionally, the weighting coefficient of the wonderful frame=1, and the formaldehyde coefficients of other image frames before or after the wonderful frame decrease in sequence.

[0037] In combination with the first aspect, in some embodiments, the electronic device may determine the repeatability of the first image based on any one or more of the scores of the first image and the historically captured images, including whether the person is moving, the subject, and the similarity.

[0038] In combination with the first aspect, in some embodiments, if any one of the first image and the historically captured images is repeated, the first image is determined to be repeated; if the first image and all of the historically captured images are not repeated, the first image is determined to be non-repeated.

[0039] In combination with any one of the above two implementations, only in any one of the following cases, the first image and the second image in the historically shot images are not repeated, and the second image is any one of the historically shot images:

[0040] The score of the first image is higher than the score of the second image;

[0041] Alternatively, the score of the first image is lower than the score of the second image, and one of the first image and the second image is a motion frame, and the other is not a motion frame;

[0042] Alternatively, the score of the first image is lower than the score of the second image, and both the first image and the second image are motion frames, and the motion time is greater than a preset minimum time interval;

[0043] Alternatively, the score of the first image is lower than the score of the second image, and both the first image and the second image are non-motion frames, and the subjects are the same, and the similarity between the first image and the second image is lower than a second threshold.

[0044] In combination with the first aspect, in some implementations, the subject of the first image includes any one of the following: children, animals, buildings, plants, and landscapes; and the subject of the first image is determined according to the content of the first image.

[0045] In conjunction with the first aspect, in some embodiments, the electronic device can determine the recommended image from the multiple first images in the following manner: divide the multiple first images into multiple theme groups according to their respective themes; sort the first images in each theme group according to the scores and repetitiveness, with the higher scores being arranged first, and non-repetitive images being arranged first compared to repetitive images; and filter the first images from each theme group according to the order of arrangement to form the recommended image. In this way, high-quality images with different themes, wonderful content, and non-repetitive images can be recommended to users.

[0046] A certain number of recommended images are selected from each theme group, which can ensure the diversity and high quality of the final recommended images.

[0047] Optionally, the number of recommended images selected by the electronic device from each theme group may be the same or different.

[0048] In combination with the first aspect, in some embodiments, before determining a recommended image from a plurality of first images based on their respective scores, themes, and repeatability, the method may further include: displaying a notification message, the notification message being used to prompt a user to view the recommended image; and detecting a user operation acting on the notification message.

[0049] In combination with the previous implementation, the notification message is displayed at any of the following time points: a preset fixed time; after a third preset number of images are taken; after entering the first location and taking multiple images at the first location, and after leaving the first location.

[0050] In conjunction with the first aspect, in some implementations, the manner of displaying the indication information of the recommended image may include the following two methods:

[0051] 1. Display thumbnails of recommended images.

[0052] 2. Displaying indication information of a plurality of first images, wherein the indication information of a recommended image is selected.

[0053] In conjunction with the first aspect, in some embodiments, after displaying the indication information of the recommended image, the method may further include: detecting a user operation for generating a video; and generating a video according to the recommended image. This can facilitate the user to generate a video with one click, meeting the user's need to make a video.

[0054] In combination with the previous embodiment, the electronic device may generate a video according to the recommended image in the following manner: matching an audio and video template according to the recommended image; and generating a video according to the recommended image and the matched audio and video template.

[0055] In combination with the previous embodiment, after generating a video according to the recommended image, the method may further include: detecting a user operation for sharing the video; sharing the video to a first application in the electronic device, or sharing the video to a first device different from the electronic device. This makes it easy for users to quickly and conveniently share videos made using the recommended image to some applications or devices, meeting the user's need to share videos.

[0056] In conjunction with the first aspect, in some embodiments, the type of any first image is any of the following: a static photo containing one image frame; a dynamic photo containing multiple image frames; a video containing multiple image frames. That is, the image recommendation method provided by the present application can recommend different types of images to the user.

[0057] In combination with the first aspect, in some embodiments, when the first image is a dynamic photo, saving the first image specifically includes: storing the moment, before and after the moment when the user operation for taking the image is detected, the preview image frame displayed on the preview interface, and the sound detected by the electronic device as a dynamic photo.

[0058] In combination with the first aspect, in some embodiments, the historical captured images specifically include any one of the following: all images taken by the electronic device before the first image; images taken after the camera was last started; images taken on the same day; images within a fourth preset number taken most recently; images taken at the first location after entering the first location.

[0059] In combination with the first aspect, in some embodiments, the multiple first images specifically include any one of the following: all images taken by the electronic device; images taken during the period from the last time the camera was started to the last time the camera was turned off; images taken on the same day; the fifth preset number of images taken most recently; images taken at the first location during the period from entering the second location to leaving the second location.

[0060] In a second aspect, an electronic device is provided, comprising: a memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute a method executed by the electronic device in the first aspect or any one of the embodiments of the first aspect.

[0061] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on an electronic device, causes the electronic device to execute a method executed by the electronic device in the first aspect or any one of the embodiments of the first aspect.

[0062] In a fourth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a computer, it enables the computer to execute the method executed by the electronic device in the first aspect or any one of the embodiments of the first aspect.

[0063] In a fifth aspect, an embodiment of the present application provides a chip system, which includes at least one processor for implementing a method executed by an electronic device in the first aspect or any one of the embodiments of the first aspect.

[0064] The image recommendation method provided in the present application is implemented to provide an intelligent image recommendation solution, which can recommend images to users based on the theme, score and repeatability of each image to meet the actual needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1A-Figure 1C A user interface for taking dynamic photos provided in an embodiment of the present application;

[0066] Figure 2A-2P A user interface for recommending images, generating videos, and sharing videos provided in an embodiment of the present application;

[0067] Figure 3A A flowchart of an image recommendation method provided in an embodiment of the present application;

[0068] Figure 3B A flowchart of another image recommendation method provided in an embodiment of the present application;

[0069] Figure 4 A flowchart for scoring the current frame preview image provided by an embodiment of the present application;

[0070] Figure 5 The face size influence coefficient s provided in the embodiment of the present application b The mapping relationship curve with face size;

[0071] Figure 6 A flowchart for determining the repeatability of two images provided in an embodiment of the present application;

[0072] Figure 7 A hardware structure block diagram of an electronic device provided in an embodiment of the present application;

[0073] Figure 8 The software architecture of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0074] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0075] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0076] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device, and finally presented as content that can be recognized by the user. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operation displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0077] The present application provides an image recommendation method and related devices.

[0078] The image recommendation method is applied to an electronic device, and the electronic device is an intelligent terminal device, which can be of various types, and the embodiments of the present application do not limit its specific type. For example, the electronic device can be a mobile phone, and can also include a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, a smart screen, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a car machine, a smart headset, a game console, and can also be an Internet of Things (IOT) device or a smart home device such as a smart TV, etc. Without limitation, the electronic device can also include a laptop computer with a touch-sensitive surface or a touch panel, a desktop computer with a touch-sensitive surface or a touch panel, and other non-portable terminal devices, etc.

[0079] In the image recommendation method, the electronic device can score multiple images separately and determine whether each image is similar. The score of an image measures the quality of the image and the degree of excitement of the content, and the similarity indicates whether an image is similar to other images. The electronic device can recommend high-quality, exciting, and dissimilar (i.e., different content) images to the user based on the scores and similarities of each image, so that the user can pay attention to these recommended images. For the detailed description of the scoring strategy and the similarity measurement method, please refer to the method implementation example below, which will not be repeated here.

[0080] In some embodiments, the electronic device may further process the recommended images. For example, the electronic device may generate a video using the recommended images to meet the user's need to produce high-quality videos with exciting content and non-repetitive content. The function of generating a video using the recommended images may be called "one-click video" or other names. For another example, the electronic device may also share the generated video to a social platform or contact to meet the user's need to share the video.

[0081] The image recommendation method provided in this application can be a function provided by the system's gallery application, or a function provided by other third-party applications, which is not limited here. System applications refer to applications provided by the manufacturer of the electronic device, and third-party applications are applications not provided by the device manufacturer.

[0082] Among them, the gallery is an image management APP on electronic devices such as smartphones and tablets, which can also be called "album" and "photo", etc. This application does not limit the name of the application. The gallery can support users to perform various operations on images stored on electronic devices, such as browsing, editing, deleting, selecting, etc. That is, the object managed by the "gallery" is the image. In other cases, the gallery can also support users to perform the above-mentioned operations on images stored on cloud servers. The gallery APP in the following embodiments of the present application also provides an image recommendation function.

[0083] For the sake of simplicity, the image recommendation method of the present application provided by the gallery is used as an example for explanation.

[0084] The images mentioned in this application may include the following categories:

[0085] 1. Live photo. Live photos include the moment of shooting and the picture content and sound of a period of time before and after the moment. That is, it contains multiple frames (such as 90 frames) of pictures and corresponding sounds, which can be played like short videos. The number of frames before and after the moment of shooting included in a live photo can be the same or different. Live photos can also be called animated pictures, live photos, etc.

[0086] 2. Still photos. Still photos include one frame of picture, which can be the picture content recorded at the time of shooting, or a frame of picture fused together from several frames before and after the shooting. Still photos only include the picture but not the sound. Both still photos and dynamic photos can be collectively referred to as photos.

[0087] 3. Video. Videos can include images and sounds recorded continuously over a period of time. Usually, the duration of a video is longer than that of a dynamic photo.

[0088] The electronic device can display a thumbnail of the image, and the original image can be stored locally on the electronic device or in the cloud. The thumbnail of a dynamic photo can be a frame captured at the time of taking the photo.

[0089] The image displayed in the electronic device may be captured by the electronic device using a camera, obtained from other applications or downloaded from the Internet, or shared by other electronic devices, without limitation here.

[0090] Before introducing the image recommendation method provided by the present application in detail, a set of UI examples on electronic devices are provided to help understand the method. The set of UIs is introduced by taking the image type of dynamic photos as an example.

[0091] Figure 1A-Figure 1C A user interface for taking dynamic photos for an electronic device.

[0092] Figure 1A An exemplary user interface 11 for displaying installed applications on the electronic device 100 is exemplarily shown.

[0093] The user interface 11 displays: a status bar, a page indicator, a tray with common application icons, other application icons, etc. Without limitation, the user interface 11 may also include a navigation bar, a sidebar, and the like. Figure 1A The user interface 11 shown may be referred to as a home screen.

[0094] The application icons in the user interface 11 may include, for example, a camera icon 111, a gallery icon 112, and icons of some other applications. The camera is an APP on an electronic device that is used to call a camera to take images. This embodiment does not limit the name of the application. The camera APP may be a system application provided by the electronic device or a third-party application.

[0095] like Figure 1A As shown, the electronic device may receive a user operation (such as a click operation or a touch operation) acting on the camera icon 111, and then may respond to the operation, start the camera application, and display a Figure 1B The user interface 12 is shown. Figure 1B The user interface 12 is a preview interface provided by the camera application.

[0096] like Figure 1B As shown, the user interface 12 may include a mode bar 121, a photo control 122, a preview window 123, a review control 124, and a quick function area 125. Among them:

[0097] The mode bar 121 may include multiple camera mode options, such as "night scene", "portrait", "photo", "video recording", etc. Different camera modes can provide users with different camera services. Users can choose any of the multiple camera modes to take pictures according to different needs. For example, "photo" can be the default camera mode, which is used to take pictures. "Video recording" is used to record videos. The "night scene" mode is suitable for photo scenes with dim light, such as at night. The "portrait" mode is suitable for photo scenes where the subject of the photo is a person. The electronic device can detect the user operation of switching modes, such as swiping left / right in the mode bar 121, and change the currently used camera mode according to the operation. For example, when it is detected that the mode bar 121 is dragged to slide to the left (left slide operation) and the float stops at the "portrait" option, the electronic device can switch to the "portrait" mode. By default, the electronic device first uses the "photo" mode.

[0098] The photo control 122 is used to trigger photo taking. The electronic device can detect whether there is a user operation on the photo control 122, such as a click operation. When the user operation on the photo control 122 is detected, the electronic device can generate a photo taking instruction. The electronic device can obtain the image reported by the corresponding camera according to the photo taking instruction, and then save it as a static photo or a dynamic photo.

[0099] The preview window 123 can be used to display the image captured by the camera of the electronic device in real time. In different shooting modes, the electronic device can process the image reported by the camera to improve the display effect of the image. For example, in the "portrait" mode, the electronic device can blur the background in the image reported by the camera to highlight the portrait. Here, the preview window 123 can display the image processed by the image processing algorithm corresponding to the different shooting modes in real time, so that the user can perceive the shooting effect corresponding to the different shooting modes in real time. The image in the preview window 123 can also be called a preview image.

[0100] The review control 124 can be used to display thumbnails of recently taken photos or videos. When a user operation acting on the review control 124 is detected, the electronic device can also display an image corresponding to the thumbnail.

[0101] The quick function area 125 may include a high-dynamic range (HDR) function option, a dynamic photo function option 125A, an AI function option, a flash function option, a color mode function option, a setting function option, etc. The HDR function option can be used to trigger the electronic device to use the HDR algorithm to fuse the image when it is turned on. The AI ​​function option can be used to trigger the electronic device to identify the photo scene in the preview screen when it is turned on, and the current AI function option is in the off state. The flash control can be used to trigger the electronic device to turn on or off the flash. The color mode function option can be used to trigger the electronic device to use a color filter to process the image captured by the camera. The setting function option can be used to set the photo parameters of the electronic device (for example, image size, image storage format, etc.).

[0102] The dynamic photo function option 125A can be used to turn on or off the dynamic photo shooting mode. Figure 1B The display status of the dynamic photo function option 125A indicates that the dynamic photo shooting mode has been turned on.

[0103] After turning on the dynamic photo shooting mode, the electronic device can detect user operations (such as click operations, touch operations, etc.) acting on the photo control 122, and then respond to the user operation to save the preview image at the photo moment and a period of time before and after the photo moment and the collected sound as a dynamic photo.

[0104] like Figure 1C As shown, after taking a dynamic photo, a thumbnail of the dynamic photo will be displayed in the review control 124. If the user clicks the review control 124 at this time, the electronic device will start the gallery and jump to the user interface that displays the dynamic photo corresponding to the thumbnail in the gallery.

[0105] according to Figure 1B-Figure 1C With the same operation, users can change the angle of the electronic device, or switch the camera, etc., to take more dynamic photos when the framing is appropriate.

[0106] Figure 2A A user interface 21 is provided for the gallery to display multiple images taken by the user.

[0107] like Figure 2A As shown, the user interface 21 displays thumbnails of multiple dynamic photos taken by the user, including thumbnails 211A to 211P. A dynamic photo indicator is displayed at the lower left corner of each thumbnail, indicating that the image corresponding to the thumbnail is a dynamic photo. Figure 2A It can be that the electronic device detects the Figure 1A The user interface displayed after the user operates on the icon 112 in the gallery.

[0108] Figure 2B-2L User interface for recommending images and generating videos for a gallery.

[0109] Figure 2B The user interface 22 shown is provided by a drop-down notification bar of an electronic device. The notification bar is a system application that provides notification messages in an electronic device. The user interface 22 displays a notification message 221 provided by the notification bar, and the specific content of the notification message 221 comes from the gallery. The notification message 221 includes: an icon 221A of the gallery, the name 221B of the gallery, the generation time 221C of the notification message, and the notification content 221D. The notification content 221D can prompt the user that the electronic device recommends images, and the user can use these images to generate videos. The notification content 221D can, for example, be "One-click blockbuster recommendation" or "Recommend you to create a wonderful short story about nature with one click."

[0110] The generation time of the notification message 221 will be described in detail in the subsequent method embodiments.

[0111] like Figure 2B As shown, after the electronic device detects the user operation acting on the notification message 221, it can jump to display Figure 2C The user interface 23 of the gallery is shown.

[0112] like Figure 2CAs shown, the user interface 23 displays: a return key 231, a title bar 232, thumbnails of multiple images taken by the electronic device within a period of time such as thumbnails 211A to thumbnail 211P, an edit bar 233, a clear control 234, and a control 235 for generating a video.

[0113] The return key 231 is used to monitor user operations. After the electronic device detects the user operation acting on the return key 231, it can respond to the user operation and display the homepage provided by the gallery for displaying images, or it can also return to the desktop.

[0114] The title bar 232 is used to indicate which images are displayed in the current user interface 23. For example Figure 2C The "all photos" in the title bar 232 refer to the photos taken by the electronic device within a period of time. There is also a drop-down arrow next to the title bar 232, and the user can click the drop-down arrow to select the image type and image range (including time range) that he wants to view from the drop-down options in the electronic device.

[0115] The images corresponding to thumbnails 211A to 211P are multiple images taken by the electronic device within a period of time. The period of time can be defined in many ways. For example, these images may include images taken from the last time the electronic device started the camera to the time the camera was turned off, or may include images taken by the electronic device on the same day, or may include a preset number of images taken recently by the electronic device, or may be images taken from the time the electronic device entered a certain place / area to the time it left the place / area, and so on.

[0116] A zoom icon may be displayed in the upper left corner of each thumbnail. The user may click on the zoom icon, and the electronic device may then display the original image of the corresponding thumbnail in full screen, making it easier for the user to view.

[0117] The lower left corner of each thumbnail can also display the image category logo, such as Figure 2C The images corresponding to the thumbnails in the image are all dynamic photos, so the dynamic photo logo is displayed in the lower left corner. If the image is a video, the video logo can be displayed in the lower left corner; if the image is a static photo, the logo may not be displayed in the lower left corner.

[0118] A check mark box may also be displayed in the lower right corner of each thumbnail. Figure 2CAs shown, the selection mark boxes of some thumbnails among the multiple thumbnails are empty, indicating that the corresponding thumbnails are not selected; the backgrounds of the selection mark boxes of some thumbnails (such as thumbnail 211D, thumbnail 211E, thumbnail 211G, thumbnail 211J, thumbnail 211K, and thumbnail 211M) are filled, indicating that the corresponding thumbnails are selected by the electronic device. The selection mark boxes of the selected thumbnails are also marked with numbers, and the numbers from small to large correspond to the order in which the images are selected. For example, the order from front to back is: thumbnail 211M, thumbnail 211K, thumbnail 211J, thumbnail 211G, thumbnail 211E, and thumbnail 211D. The original image corresponding to the selected thumbnail is the image recommended by the electronic device to the user. For specific recommendation strategies, please refer to the introduction of the method embodiment below.

[0119] The electronic device can detect the user operation acting on the blank selection mark box and mark the corresponding thumbnail as being selected.

[0120] Some of the thumbnails selected by the electronic device are displayed in the edit bar 233. The electronic device can detect the left / right swipe operation on the edit bar 233, and in response to the operation, switch to display other thumbnails selected by the electronic device. The lower left corner of each thumbnail in the edit bar 233 can also display an image category logo.

[0121] The edit bar 233 can be used to edit the thumbnails selected by the electronic device, and the editing operation includes, for example, deletion, order adjustment, etc. For example, a delete control is displayed in the upper right corner of each thumbnail in the edit bar 233, and the electronic device can detect the user operation on the delete control of the thumbnail, and then respond to the user operation, mark the corresponding thumbnail as unselected, and remove it from the edit bar 233. For another example, the electronic device can also detect the user operation of long pressing the thumbnail in the edit bar 233 and then dragging it to other positions in the edit bar 233, and then respond to the user operation to adjust the order in which the corresponding thumbnails are selected, such as advancing or placing them in the back.

[0122] The clear control 234 can be used to monitor user operations. After the electronic device detects the user operation acting on the clear control 234, it can respond to the user operation and mark all previously selected thumbnails as unselected, thereby achieving the purpose of clearing the selection.

[0123] The number of thumbnails currently selected by the electronic device may be displayed on the control 235 , and the number may change as the user adds or deletes selected thumbnails.

[0124] The control 235 may be used to monitor user operations. After the electronic device detects the user operation acting on the control 235, it may respond to the user operation and synthesize the original image corresponding to the selected thumbnail into a video.

[0125] If the electronic device enables the "one-key photo" function for the first time, the electronic device may also display the following after detecting the user operation on the control 235: Figure 2D The user interface 24 is shown. The user interface 24 is used to obtain authorization for the relevant permissions required for the "one-click filming" function.

[0126] like Figure 2D As shown, the user interface 24 displays: prompt information 241 , prompt information 242 , cancel control 243 , and agree control 244 .

[0127] Prompt information 241 is used to prompt the user to enable the service or application that will be started after the “one-click to create a picture” function is turned on, that is, the service or application that supports the implementation of the “one-click to create a picture” function. Figure 2D The "editing" service mentioned in the video provides functions such as video capture, adjustment, and trimming. The "editing" service can also be called other names, which are not limited here. The "editing" service can be a system service or a third-party service.

[0128] Prompt information 242 is used to prompt the user to obtain the user authority required to start the "Clip" service. For example, Figure 2D It indicates that the use of the "Clip" service requires network permission. The prompt information 242 may also provide some controls for the user to click to view more detailed instructions.

[0129] The cancel control 243 is used to monitor user operations. After the electronic device detects the user operation acting on the cancel control 243, it can respond to the user operation and return to the display. Figure 2C The user interface 23 is shown.

[0130] The consent control 244 is used to monitor user operations. After the electronic device detects the user operation acting on the consent control 244, it can respond to the user operation. The electronic device can obtain the user authorization to use the "Clip" service, and thus can use the "Clip" service via the network.

[0131] After the electronic device detects the user operation acting on the consent control 244, it can display Figure 2E The user interface 25 shown, Figure 2F User interface 26 is shown.

[0132] like Figure 2EAs shown, the user interface 25 displays prompt information for prompting the user to grant the "Clip" service access to music and audio in the electronic device, as well as a prohibition authorization control and an always allow control. After the electronic device detects that the always allow control is activated, the music and audio access rights can be permanently granted to the "Clip" service.

[0133] Electronic devices detect the action on Figure 2E After always allowing controls in, such as Figure 2F As shown, the user interface 26 displays prompt information for prompting the user to grant the "Clip" service access to photos and videos in the electronic device, as well as a prohibition authorization control and an always allow control. After the electronic device detects that the always allow control is activated, the access rights to photos and videos can be permanently granted to the "Clip" service.

[0134] Electronic devices detect the action on Figure 2F After you set the Always Allow control, the electronic device can start compositing the video.

[0135] Of course, if this is not the first time that the electronic device is used, and if this is not the first time that the "One-click Photo" function is enabled, there is no need to display Figure 2D-2F The user interface shown in the figure detects the Figure 2C After the user operates the middle control 235, if the user authorization has been obtained, the video synthesis can be started directly.

[0136] The specific process of synthesizing video includes the following:

[0137] First, the original image (also referred to as material) corresponding to the thumbnail selected by the electronic device (i.e., the thumbnail included in the edit bar 233) is analyzed, for example, various information thereof is obtained, such as size, quantity, theme, etc. During the analysis process, the electronic device may display Figure 2G The user interface 27 shown in FIG. The user interface 27 displays: a prompt message 271 for prompting the user that the electronic device is analyzing the material; a progress bar and progress value 272 for prompting the user of the progress of the analysis; and a stop control 273 for interrupting the analysis operation and the subsequent operation of generating a video.

[0138] After the analysis is completed, the video template is matched to the selected original image. During the matching process, the electronic device can display Figure 2H The user interface 28 shown in FIG. The user interface 28 displays: prompt information 281 for prompting the user that the electronic device is matching materials; a progress bar and progress value 282 for prompting the user of the matching progress; and a stop control 283 for interrupting the matching operation and the subsequent video generation operation.

[0139] The specific process of analysis and matching can be referred to the detailed description of the method embodiment below, which will not be described here.

[0140] After the matching is completed, the video generation begins. During the video generation process, the electronic device can display Fig.2I User interface 29 is shown.

[0141] like Fig.2I As shown, the user interface 29 displays: a return key 291, a share key 292, a prompt message 293, a play / pause control 294, a video progress bar 295, and a menu bar 296 at the bottom.

[0142] The back key 291 can be used to return to Figure 2C The user interface 23 is shown.

[0143] The share button 292 can be used to share the finished video to other applications or contacts.

[0144] The prompt information 293 is used to prompt the user that a video is currently being generated. After the electronic device has finished making the video, the video can be automatically played in the area 297 where the prompt information 293 is located.

[0145] The video progress bar 295 is used to indicate the playing progress and the total length of the video. Before the video is made, the playing progress and the total length of the video can both be 0.

[0146] The menu bar 296 includes a plurality of controls, such as:

[0147] Template control 296A, which the user can click to view more video templates and select an appropriate template to synthesize the previously selected original image;

[0148] Music control 296B, which the user can click to view optional music and synthesize it into the video;

[0149] The export control 296C is used to export the generated video, that is, store it as a video file; before the video is generated, the export control 296C may be in a non-clickable state;

[0150] Segment controls 296D that the user can click to view individual segments in the generated video;

[0151] Editing control 296E is used to edit the generated video, such as cropping, adding filters, etc.

[0152] refer to Figure 2J , Figure 2J The diagram exemplarily shows a frame of user interface 30 displayed when an electronic device automatically plays a video after the video is produced.

[0153] If the user decides to save the video after watching it in area 297, he or she may input a user operation into the export control 296C in the user interface 30, and the electronic device may respond to the operation by exporting the video into a certain video format and storing it in the storage area of ​​the electronic device.

[0154] Figure 2K The user interface 31 displayed during the process of exporting a video by an electronic device is exemplarily shown. Figure 2K As shown, the user interface 31 displays: prompt information 311, used to prompt the user that the electronic device is exporting the video; a progress bar and progress value 312, used to prompt the user the progress of exporting the video; and a stop control 313, used to interrupt the operation of exporting the video.

[0155] After the video is exported, the electronic device can display Figure 2L The user interface 32 is shown. The user interface 32 displays a prompt message 321, which is used to prompt the user to which album in the gallery the generated video has been saved, and that the video has been saved to the template draft of the "Clip" service. That is, after the electronic device exports the video, in addition to storing it in the album storage path under the gallery, it can also store it in the storage path of the template draft under the "Clip" service.

[0156] The prompt information 321 may disappear automatically after being displayed for a period of time, or may disappear in response to a user operation (for example, the user clicks Figure 2L The electronic device may then continue to display similar Figure 2J User interface.

[0157] After storing the video, the electronic device can view and share the video in the corresponding album of the gallery.

[0158] Figure 2M-2P A user interface for viewing videos in the gallery and sharing videos.

[0159] Figure 2M The user interface 33 of the gallery displaying the album in the electronic device is shown. The album here refers to the presentation of the image classification results in the gallery. Figure 2M As shown, the user interface 33 displays a thumbnail 331 of the “Video” album and the number of videos contained in the album.

[0160] After the electronic device detects a user operation on a thumbnail of the "Video" album, it can display Figure 2N The user interface 34 shown is used to display the thumbnails and video duration of each video contained in the "Video" album. Figure 2N As shown, the user interface 34 displays thumbnails 341 of videos previously generated by the electronic device and their corresponding durations.

[0161] Electronic devices can detect the Figure 2N The user operation of the video thumbnail 341 generated by the electronic device is then displayed Fig.2O The user interface 35 shown in the figure plays the video in the video playing area 351 of the user interface 35. The user interface 25 also displays various operation controls for the video, such as a progress bar, a sharing control 352, a collection control 353, an editing control 354, a deletion control 355, and the like.

[0162] like Fig.2O As shown, the electronic device can detect the user operation acting on the sharing control 352, and then display the Figure 2P The user interface 36 shown. The user interface 36 displays any one or more of the following: one or more application options, one or more contact options, one or more device options, etc. Among them, an application option corresponds to an application, which is used to share the video to the application; the contact option corresponds to the contact of the electronic device, which is used to share the video to the contact's device; the device option corresponds to the nearby device discovered by the electronic device, which is used to share the video to the nearby device.

[0163] For example, Figure 2P As shown, after the electronic device detects the user operation acting on the application option 361, it responds to the operation and shares the currently displayed video to the application corresponding to the application option 361. Of course, the user can also share the video to other applications, other devices, or other contacts according to actual needs.

[0164] Figure 3A The following is an exemplary diagram of a process of an image recommendation method provided by the present application. The method is executed by an electronic device.

[0165] like Figure 3A As shown, the method may include the following steps:

[0166] S101, an electronic device captures a plurality of images.

[0167] The images captured by the electronic device may be static photos, dynamic photos or videos, and there is no limitation here.

[0168] For the specific process of electronic devices taking multiple images, please refer to the following Figure 3B S301-S304 in the method shown.

[0169] S102, the electronic device determines the subject, score, and repeatability of each captured image.

[0170] The score of each image can be obtained by the electronic device according to any one or more of the image quality, composition, facial beauty, and task action of the image. Repeatability refers to whether an image is repeated in the historical images taken before the image. Theme refers to the scene reflected by the image.

[0171] For details on how electronic devices can determine the subject, score, and repetitiveness of an image, see Figure 3B S305 in.

[0172] S103: The electronic device determines a recommended image from the plurality of images according to the scores, themes and repetitiveness of the plurality of images respectively.

[0173] For the specific implementation of S103, please refer to Figure 3B S306-S310 in.

[0174] S104, the electronic device displays indication information of the recommended image.

[0175] For the specific implementation of S104, please refer to Figure 3B S311 in.

[0176] Figure 3B Another process of the image recommendation method provided by the present application is exemplarily shown. The method is executed by an electronic device.

[0177] like Figure 3B As shown, the method may include the following steps:

[0178] S301, the electronic device starts the camera and displays a preview image.

[0179] The electronic device can run system applications or third-party applications to start the camera to capture images, receive the preview stream reported by the camera, and display the preview image in the preview interface displayed on the display screen based on the preview stream. The camera started by the electronic device can be any camera, depending on user needs.

[0180] After the electronic device starts the camera, it can continuously receive the preview stream reported by the camera, so the preview image displayed by the electronic device will also include continuous multiple frames (ie, multiple preview image frames).

[0181] For example, reference 1A- Figure 1B , the electronic device can run the camera application and Figure 1B The preview image frame is displayed in the preview window 123. Figure 1B The illustrated user interface 12 may be referred to as a preview interface.

[0182] S302, analyzing the current frame preview image from the preview stream, and obtaining a tag of the frame preview image, the tag including the following information: score, theme, and whether it is a transition frame.

[0183] The current preview frame image refers to a preview image frame currently displayed on the display screen. The current preview frame image may also be referred to as a preview image frame.

[0184] The electronic device can analyze the current preview frame image from the preview stream.

[0185] When scoring a preview image, any one or more of the following factors may be considered:

[0186] 1. Image quality.

[0187] Image quality includes image clarity and / or symmetry. The higher the clarity and / or the better the symmetry, the higher the image quality and the higher the image score. Symmetry refers to the degree of symmetry of the image content (such as the left-right symmetry of a building), which can be reflected by a score of [0, full score].

[0188] 2. The beauty of the face.

[0189] Each frame of an image includes a subject, and the subject of the image refers to an object of a category such as a person or an animal that occupies a relatively large proportion in the picture. A frame of an image may include one or more subjects. An electronic device may use a subject detection algorithm to detect and identify the subject category in an image, and the subject detection algorithm may be a convolutional neural network (CNN).

[0190] If the subject of the image is a person or an animal, the image may be scored based on the aesthetics of the face of the person or animal in the image.

[0191] The more beautiful the face in the image is, the higher the score of the image will be.

[0192] 3. The movements of the characters in the image.

[0193] If the subject of the image includes a person, the image can be scored according to the excitement of the person's movements. If the person has a moving movement and the movement is a preset exciting movement, the image can be considered exciting. The preset exciting movement movements may include, for example, jumping, running, raising hands, raising legs, etc.

[0194] The more exciting the character's movements in the image, the higher the image score.

[0195] 4. Composition.

[0196] Composition refers to the arrangement, planning, and position of objects in an image. The more coordinated and complete the arrangement and position of objects, the more beautiful the composition will be, and the higher the image score will be.

[0197] When scoring the current frame preview image according to any one or more of the above four factors, the weight of each factor on the final score can be the same or different. The weight of each factor on the final score can be preset according to actual needs and is not specifically limited here.

[0198] The following takes the above four factors into consideration as an example to explain how to score the preview image of the current frame.

[0199] Formula 1 exemplarily shows a formula for calculating the preview image of the current frame by the electronic device:

[0200] score=a1*score1+a2*score2+a3*score3+a4*score4 Formula 1

[0201] Among them, score is the score of the preview image of the current frame, score1 is the score in the dimension of image quality, score2 is the score in the dimension of composition, score3 is the score in the dimension of facial beauty, and score4 is the score in the dimension of character action. The higher the score, the better the factor of the corresponding dimension. score1, score2, score3, and score4 can also be called the first score, the second score, the third score, and the fourth score, respectively.

[0202] The scoring system of each dimension is consistent. The scoring system can be any scoring system, such as a percentage system, a ten-point system, etc. The score interval of the scoring system, such as [0, 10] of a ten-point system and [0, 100] of a percent system, can be called a first score interval. The highest score in the first score interval, such as 10 points of a ten-point system and 100 points of a percent system, can be called a full score.

[0203] In the various implementations of the scoring process introduced below, a high score includes a score whose difference from the full score is less than a preset value, such as 9-10 points on a 10-point scale, such as 90-100 points on a 100-point scale, etc.; a low score includes a score whose difference from the full score is greater than a preset value, such as 0-2 points on a 10-point scale, such as 0-20 points on a 100-point scale, etc.

[0204] a1, a2, a3, and a4 are the weights of image quality, composition, facial beauty, and character action on the final score. The sum of a1, a2, a3, and a4 is 1.

[0205] refer to Figure 4 , Figure 4The process of scoring the preview image of the current frame by the electronic device is shown as an example. The process includes the following steps:

[0206] Step 1: Set the initial score1, score2, score3, and score4 to 0.

[0207] Step 2: Calculate the image quality score score1 of the current frame preview image.

[0208] score1 can be calculated based on an image quality assessment (IQA) algorithm.

[0209] Step 3, determine whether the current frame preview image contains a subject, if not, go directly to step 9; if contained, go to step 4.

[0210] Step 4: Calculate the composition score score2 of the current frame preview image.

[0211] score2 can be calculated based on a composition scoring algorithm. The composition scoring algorithm can be used to score images with a subject, and to evaluate whether the composition of the subject in the overall picture is beautiful. The composition scoring algorithm can be a calculation method based on human skeleton nodes.

[0212] Step 5, determine the category of the image subject, if it is an animal, go to step 6; if it is a person, go to step 7.

[0213] Step 6, if an animal's face is detected, the facial beauty score score3 is set to a high score, which can be preset, for example, a full score (such as 100 points on a 100-point scale) or a value close to the full score (such as 90-100 points on a 100-point scale).

[0214] For animals, as long as the face is captured, the image content is acceptable to the user because a high score can be given for the facial beauty. In a specific embodiment, a high score can be given for the facial beauty when the front face of the animal is detected.

[0215] A high score can be set as a score whose difference from the full score is less than a preset value.

[0216] Step 7, calculate the facial beauty score score3 of the character.

[0217] Usually users have high requirements for the beauty of human faces, so a more detailed calculation method can be used. For example, score3 can be calculated based on the expression of the person in the image, the posture of the face, the proportion of the face in the image, the number of faces, etc. The expression of the person may include whether the person has open eyes or a smile, etc., and the facial posture includes whether the person is facing the camera.

[0218] In one implementation, score3 may be calculated based on Formula 2:

[0219]

[0220] In formula 2,

[0221] score3 is the total score of facial beauty, N face is the number of faces, s e,i is the eye-opening and eye-closing score of the i-th face, s m,i is the smile score of the i-th face, s d,i is the posture correctness score of the i-th face, s b is the face size influence coefficient, s N is the face ratio coefficient when there are multiple faces, s c is the composition beauty coefficient, s p is the face-free penalty threshold.

[0222] Among them, the score of eyes open and closed is s e,i , Smiley score m,i , Correct posture score d,i The scoring range of is within the scoring range of the facial beauty score score3. For example, if score3 is scored in percentage, the scores of the above three factors are also within the interval [0,100].

[0223] Face size influence coefficient s b , face ratio coefficient s when there are multiple faces N , composition beauty coefficient s c , no face penalty threshold s p The values ​​are all in the interval [0, 1].

[0224] The following is an introduction to the various parameters in formula 2:

[0225] Eyes open and closed score e,i : If the character's expression is considered beautiful when the character opens his eyes, the first value is assigned; if the character's expression is considered ugly when the character closes his eyes, the second value is assigned, and the first value is higher than the second value. The first value can be a high score, and the second value can be a low score.

[0226] Smiley score m,i: A smiling person is considered to have a beautiful expression and is assigned the third value; a person without an expression is considered to have an ugly expression and is assigned the fourth value. The third value is higher than the fourth value. The third value can be a high score and the fourth value can be a low score.

[0227] Correct posture score d,i : When the face is facing the camera, the posture is considered to be correct and the fifth value is assigned; when the face is facing the camera, such as looking up or lowering the head, the face posture in the picture is considered to be incorrect and the sixth value is assigned. The fifth value is higher than the sixth value. The fifth value can be a high score and the sixth value can be a low score. Face size influence coefficient s b : The face area of ​​appropriate size (such as the face area within the first range) is assigned the seventh value. When the area is too large (such as the face area outside the first range), the face is incomplete and is considered unsightly. When the area is too small, the non-subject (possibly a passerby) is photographed as background interference and is considered unsightly. When it is unsightly, the face size affects the coefficient s. b The eighth value is assigned, and the seventh value is higher than the eighth value. The seventh value can be a higher value in the value range, and the eighth value can be a lower value in the value range.

[0228] Face size influence coefficient s b Can be based on Figure 5 The three Gaussian curves shown are obtained by query. Specifically, it is calculated based on the part above the intersection of the three Gaussian curves. The horizontal axis is It represents the square root of the ratio of the face frame area to the image area, and the vertical axis is the face size influence coefficient s b The three curves from left to right represent the face size at far, medium and close distances and the face size influence coefficient s b This can combine the subjective evaluation of faces at different distances and limit the scores of faces that are too large or too small.

[0229] Among them, the three Gaussian curves from left to right are:

[0230] μ=0.1278,sigma=9.5Long distance

[0231] μ=0.3819,sigma=3.5 medium distance

[0232] μ=0.65,sigma=9.5Close distance

[0233] Figure 5 The three lines shown represent the face size at far, medium and near distances and the face size influence coefficient s respectively. bThe Gaussian curve of the mapping relationship is preset. Among them, the parameter μ represents the mean, sig represents the inverse of the standard deviation, and the three lines indicate that the subject has an optimal composition coefficient at near, medium and far distances. The peak of the curve represents the best position. Using the position above the intersection can ensure that the coefficient value within a reasonable distance maintains a high score. w refers to the face frame width, h refers to the face frame height, W refers to the image width, and H refers to the image height. The face frame refers to a frame obtained by framing the face position with a rectangle or square when the electronic device analyzes the preview image of the current frame.

[0234] When there are multiple faces, the ratio of each face is s N : According to the face area from large to small, it is considered that the influence of each face on the overall picture decreases in turn. Face ratio coefficient s of multiple faces N is the preset weight, the sum of which is 1. That is, several face ratio coefficients s are preset in descending order. N , and then correspond to each face according to the area of ​​each face in the image.

[0235] Composition beauty coefficient The main character requires that the face is within a certain range (such as the first area) in the center of the picture; when the face is at the edge, it is considered not to be the main character, and the face is incomplete and the composition is considered unsightly, so the face will be multiplied by the penalty coefficient k to reduce the score of the face. The penalty coefficient k is a preset value. The ninth value is higher than the tenth value. The ninth value can be a higher value within the value range, and the tenth value can be a lower value within the value atmosphere.

[0236] Penalty threshold s when there is no face p : A human body is detected but a face is not detected. It may be that the face is not in the picture or the person is facing away. This is considered unsightly and the score of the corresponding face beauty is reduced by score3.

[0237] According to the calculation method of formula 2 above, score3 is higher when the face posture is correct, the eyes are open and smiling, and the area is moderate, and score3 is higher when the characters in the picture are all main characters.

[0238] The high scores and low scores mentioned above are relative concepts and can be set by the electronic device according to actual needs. This application does not limit their specific values.

[0239] In some implementations, if the subject of the current frame preview image includes both people and animals, the algorithms of step 6 and step 7 may be combined to calculate a comprehensive facial beauty score score3 using weights.

[0240] After step 7, you can go to step 8.

[0241] Step 8, calculate the character action score score4.

[0242] If the person in the image does not have any exciting movements, score4 is set to a low score (eg, 0).

[0243] If the person in the image has a wonderful action, the confidence output by the action evaluation algorithm can be multiplied by a high score (e.g., full score) to obtain a value, which can be set as the value of score4. The action evaluation algorithm is an algorithm that evaluates the action of the person in the image and the probability of the action, which can be a CNN algorithm. The confidence is the probability that the action of the person in the image output by the action evaluation algorithm is a certain action. The value of score4 is in the range of [0, full score].

[0244] Step 9: Calculate the score of the preview image of the current frame based on Formula 1.

[0245] based on Figure 4 The method shown is used to score the current frame preview image, and the score of the current frame preview image can reflect the quality of the image, the beauty of the face in the image, the excitement of the character's movements, the composition, etc. The higher the score, the better the quality of the image, the beauty of the face in the image, the excitement of the character's movements, the composition, etc.

[0246] Of course, it is not limited to considering Figure 4 In some other implementations, only some of the four factors in the process shown may be considered to score the preview image of the current frame. If only some of the factors are considered, the weight of each factor may change, and the sum of the weights is always 1.

[0247] For example, in the above step 5, if the current frame preview image does not include subjects such as animals and people, such as when the picture is a landscape, the weight coefficient a3 corresponding to score3 in formula 1 is set to 0, and the remaining weights a1, a2, and a4 are increased accordingly.

[0248] In some embodiments, the electronic device may further analyze whether the current frame preview image is a transition frame. When the user moves the electronic device quickly, the preview stream reported by the camera of the electronic device is a transition frame, and the imaging information amount of the transition frame is low and the image quality is poor, that is, the image frame whose information amount and image quality do not reach the first threshold. The electronic device may use a transition detection algorithm to identify the transition frame.

[0249] In some implementations, the electronic device may also analyze the subject of the current frame preview image. The subject refers to the scene contained in the image, for example, it may include children, animals, buildings, plants, landscapes, etc. The subject may be obtained based on the content analysis of the image.

[0250] In some embodiments of the present application, the electronic device may use a perception engine to score the current frame preview image, and analyze whether it is a transition frame and its theme, etc. The perception engine may be equipped with the above-mentioned various algorithms to calculate the required results.

[0251] S303: perform image encoding on the current frame preview image and save the label of the current frame preview image.

[0252] Image coding, also known as image compression, refers to the technology of representing an image or the information contained in the image with fewer bits under the condition of meeting certain quality requirements (such as signal-to-noise ratio requirements). This can save storage space of electronic devices.

[0253] In some implementations, the electronic device may encode the tag of the current frame preview image separately into the audio stream of the current frame preview image, and may store it separately from the picture data.

[0254] Before the user inputs the command to capture an image, the camera of the electronic device will continuously report the preview stream, and during this process the electronic device will continue to display multiple frames of preview images. Therefore, the above steps S302-S303 will be continuously executed multiple times, and the electronic device can save compressed images and labels of multiple frames of preview images. This part of the saved data can be temporarily stored in the electronic device, and is stored in a different location from the data saved when the image is actually captured later.

[0255] In some implementations, the electronic device may only save a preset number of compressed images and labels of preview images, and compressed images and labels of preview images with earlier generation times exceeding the preset number may be discarded. This does not affect the subsequent implementation of the present solution and can also save storage space of the electronic device.

[0256] Through the above S302-S303, the tag information of the preview image is analyzed when the image is previewed, which can save the execution time of subsequent steps and reduce the time cost. In addition, since the storage capacity of the preview stream is small, this also facilitates the convenience of the calculation and analysis process.

[0257] S304: Detect a user operation for capturing an image, and save a corresponding image in response to the user operation.

[0258] User actions for capturing images can be divided into the following categories:

[0259] 1. When the electronic device starts the dynamic photo function, a photo operation is input. In this case, the electronic device stores the preview stream as a dynamic photo. The photo operation may take many forms, such as a touch operation on the display screen, a voice command, a specific expression, etc. For example, refer to Figure 1BThe photo-taking operation may be a user operation on the photo-taking control 122 when the dynamic photo function of the electronic device is activated.

[0260] For dynamic photos, in addition to analyzing the labels of the preview images before the user operation of taking the image in S302-S303, it is also necessary to analyze the labels of the preview stream images reported by the camera within a period of time after the user operation. The specific analysis method is referred to above. That is, the electronic device needs to analyze and obtain the labels of each frame image in the dynamic photo.

[0261] 2. When the electronic device does not start the dynamic photo function, a photo operation is received. In this case, the electronic device stores the preview stream as a static photo. The photo operation can be in various forms, which are not limited here.

[0262] 3. After the electronic device starts the "recording" mode, the received operation of starting recording and the operation of ending recording will store the preview stream during this period as a video.

[0263] It can be seen that the image saved by the electronic device may include one or more preview image frames displayed on the preview interface.

[0264] The image capturing operation in S304 may also be performed multiple times, so that the electronic device can save multiple images according to the needs of the user. The multiple images saved by the electronic device may include one or more types, such as dynamic photos, static photos and videos.

[0265] For example, reference Figure 2A The user interface 21 displays thumbnails of multiple dynamic photos taken by the electronic device, including thumbnails 211A to thumbnails 211P.

[0266] Not limited to the electronic device shown in S301-S304 taking images through its own camera, in some other embodiments of the present application, the electronic device can save images sent by other applications other than the camera, or download images from the network, or receive images shared by other electronic devices, and perform subsequent steps based on these images.

[0267] The image saved by the electronic device may be referred to as a first image.

[0268] S305, obtaining a tag of the stored image, the tag including the following information: subject, score, and whether it is a repeated frame of the stored image.

[0269] The subject of the saved image

[0270] For dynamic photos, the subject that appears most frequently in multiple frames of the dynamic photo is used as the subject of the dynamic photo. If there are multiple subjects with the highest frequency, the subject with the highest priority can be selected from them according to the preset subject priority and used as the subject of the dynamic photo.

[0271] For a static photo, the subject of a frame contained in the static photo is the subject of the static photo.

[0272] For videos, the method of confirming the subject is the same as that of dynamic photos. Please refer to the relevant description.

[0273] Fraction of images saved

[0274] After saving the image, the electronic device will calculate a score for the image.

[0275] The score of the saved image is measured by one or more frames contained in it. The higher the score of the one or more frames contained in it is, the higher the score of the saved image is.

[0276] For dynamic photos, an exemplary scoring method is provided: traverse the image queue in the dynamic photo and find all the frames it contains; optionally, remove the transition frames to obtain the remaining frame sequence FrameSeq [frameStart, ..., frameEnd]. This step can ensure the validity of the final score; then find the frame with the highest score from FrameSeq and record it as the wonderful frame WonderfulFrame, and look for K frames before and after the wonderful frame. If there are less than K frames, fill them with a score of 0; set the weight of the wonderful frame to 1, and the weights of the K frames before and after are reduced by 0.1. Calculate the weighted average score weightAvgScore of the 2K+1 frame scores as the scoring result of the dynamic photo. Remove the transition frames to obtain the image frames contained in the remaining frame sequence FrameSeq, which can also be called the first preview image frame.

[0277] Of course, the above is only an example, and adjustments can be made based on the method of the above example. For example, a K frame can be found before the wonderful frame, a K' frame different from K can be found after the wonderful frame, and a weighted average score of 1+K+K' frames can be calculated.

[0278] Equivalently, the electronic device can determine the score of the saved image by taking the weighted average of the wonderful frame, the first preset number of preview image frames before the wonderful frame in the remaining frame sequence FrameSeq, and the second preset number of preview image frames after the wonderful frame in the remaining frame sequence FrameSeq.

[0279] For a static photo, the score of a frame contained in the static photo is the score of the static photo.

[0280] For videos, the way to confirm their scores is the same as that for dynamic photos. Please refer to the relevant description.

[0281] Repeatability of saved images

[0282] If there is a historical image that is repeated with the saved image, the saved image is considered to be repeated; if all historical images and the saved image are not repeated, the saved image is considered to be non-repeated.

[0283] The number of historical images may be one or more. The definition of historical images may include any of the following:

[0284] 1. All images stored in the electronic device, such as all images in the gallery.

[0285] 2. The image captured by the electronic device after the camera was last activated.

[0286] 3. Images taken by electronic devices on the same day.

[0287] 4. Images within a fourth preset number recently taken by the electronic device.

[0288] 5. Images taken by an electronic device at a certain place / area after entering the place / area.

[0289] The above are only several exemplary definitions of historical photographed images and should not constitute limitations. The electronic device in this application can define historical photographed images according to actual needs, and no limitation is made here.

[0290] The following is an exemplary method for determining the repeatability of any one of the saved images and the historical images:

[0291] refer to Figure 6 , Figure 6 The process of making a repeatability judgment on two dynamic images is exemplified. The process may include the following steps:

[0292] Step 1: Set the initial similarity score similarScore to 0.

[0293] Step 2, determine whether a.imageScore is greater than b.imageScore, if so, go to step 3; if not, go to step 4.

[0294] a represents the image currently saved by the electronic device, and b represents an image in the historical images. a.imageScore is the score of the image saved by the electronic device, and b.imageScore is the score of an image in the historical images.

[0295] Step 3, set similarScore to 0.

[0296] In this way, image a with a higher score can be marked as a non-repeated frame, so that image a with a higher score is more likely to be recommended later, so that the recommendation result is of high quality.

[0297] Step 4, determine whether one of image a and image b is a motion frame and the other is not a motion frame. If so, go to step 5; if not, go to step 6.

[0298] Whether an image is a motion frame may be determined based on a motion evaluation algorithm.

[0299] Step 5, set similarScore to 0.

[0300] Step 6, determine whether both image a and image b are motion frames, if so, go to step 7; if not, go to step 10.

[0301] Step 7: Is the time interval between the motion frames in image a and image b less than the minimum interval intervalMin? If yes, go to step 8; if no, go to step 9. The minimum interval intervalMin can be preset as needed.

[0302] Step 8, set similarScore to 1.

[0303] Step 9, set similarScore to 0.

[0304] Step 10, determine whether the subjects in image a and image b are the same, if so, go to step 11; if not, go to step 12.

[0305] The same subject here not only refers to the same category, but also means that the subject is the same, such as the same person, the same animal, etc.

[0306] Step 11, set similarScore to 1.

[0307] Step 12, using any one of the mean hash algorithm, Euclidean distance algorithm, cosine distance algorithm, Hamming distance algorithm, etc. to calculate the similarity between image a and image b, and set similarScore to the calculated value.

[0308] Step 13, determine whether similarScore is greater than a similarity threshold threshold, if so, go to step 14; if not, go to step 15. The similarity threshold threshold can be preset as needed.

[0309] Step 14, determining to save the image as a duplicate image.

[0310] Step 15, determining that the saved image is a non-duplicate image.

[0311] Since the electronic device can store multiple images, S305 can also be executed multiple times to calculate the scores of the multiple stored images respectively.

[0312] In some implementations, the electronic device may encode the tag for saving the image separately into the audio stream of the image, and may store it separately from the picture data.

[0313] S306: Detecting a user operation for acquiring a recommended image.

[0314] In some embodiments, the electronic device may generate and display a notification message, which may be used to prompt that an image can be recommended, or that a video can be created. The notification message may be sent by the gallery application to the notification application, and ultimately displayed by the notification application. The user operation for obtaining the recommended image may include a user operation acting on the notification message (such as a click operation, a touch operation, etc.).

[0315] For example, reference Figure 2B The notification message 221 in the user interface 22 is a possible implementation form. Not limited to displaying the notification message in the pull-down notification bar, the electronic device can also display the notification message at the top of the display screen after generating it, and if there is no user interaction with it, it can automatically disappear after staying for a period of time.

[0316] The generation and display time of the notification message may include any of the following:

[0317] 1. Generate notification messages periodically, such as automatically generating at 4:00 p.m. or 9:00 p.m. every day, or generating at a fixed time every week. That is, a fixed message can be generated and displayed at a preset fixed time.

[0318] 2. Generate a notification message after the electronic device has newly saved a third preset number of images.

[0319] 3. When the electronic device enters a certain place / area, captures a plurality of images in the place / area, and then leaves the place / area, a notification message is generated. The place / area may be referred to as the first place.

[0320] In other embodiments, a “one-click recommendation” control may be displayed in the user interface of an application that provides an image recommendation function (such as a gallery application or other application), and the user can input a user operation on the “one-click recommendation” control to trigger subsequent steps. The user operation acting on the “one-click recommendation” control detected by the electronic device is the user operation for obtaining a recommended image. The embodiments of the present application do not limit the user interface in which the “one-click recommendation” control is provided and the form of the “one-click recommendation” control. The name of the “one-click recommendation” control is only an example, and its function has been clearly explained. The control may also be called by other names.

[0321] S307, traverse multiple images and obtain a label of each image.

[0322] The multiple images here can include any of the following:

[0323] 1. All images stored in the electronic device, such as all images in the gallery.

[0324] 2. Images captured by the electronic device from the time the camera was last turned on to the time it was last turned off.

[0325] 3. Images taken by electronic devices on the same day.

[0326] 4. The fifth preset number of images most recently taken by the electronic device.

[0327] 5. Images captured by an electronic device between entering a certain place / area and leaving the place / area.

[0328] The electronic device may extract the labels of the respective images from the memory.

[0329] For example, reference Figure 2A Assuming that thumbnails 211A to 211P are images captured by the electronic device from the last time the camera was turned on to the last time the camera was turned off, the electronic device can extract tags of the original images corresponding to these thumbnails.

[0330] S308, grouping the multiple images traversed in S307 according to themes.

[0331] Exemplarily, if the multiple images include original images corresponding to thumbnails 211A to 211P, the multiple images may be divided into the following groups:

[0332] Landscape group: thumbnail 211A, thumbnail 211B, thumbnail 211F, thumbnail 211J, thumbnail 211L, thumbnail 211N, thumbnail 211O, thumbnail 211P, and corresponding original images.

[0333] Character group: thumbnail 211G, thumbnail 211H, thumbnail 211I, thumbnail 211K, thumbnail 211M, and corresponding original images.

[0334] Cat group: thumbnail 211E, corresponding original image.

[0335] Dog group: thumbnail 211C, thumbnail 211D, and corresponding original images.

[0336] S309, sorting the images in each group according to the repeatability and the score.

[0337] In some embodiments, for each group of images, non-repeated images are arranged first and repeated images are arranged last; then, the non-repeated images are sorted from high to low according to the scores, and the repeated images are also sorted from high to low according to the scores.

[0338] In other implementations, for each group of images, they are first sorted from high to low according to the scores. When there are images with the same scores, the non-duplicate images are placed in front and the duplicate images are placed in the back.

[0339] S310, selecting images with the highest ranking from each group of images as recommended images.

[0340] In some implementations, the electronic device may select the top K images from different theme groups to form a recommendation sequence. This can ensure the diversity and high quality of the recommended images. K can be preset, and its specific value is not limited here.

[0341] In other embodiments, the electronic device may also select different numbers of top ranked images from different theme groups to form a recommendation sequence. For example, the electronic device may select a certain proportion of the actual number of images in each theme group to form a recommendation sequence.

[0342] In some further implementations, the electronic device may also select images from only part of the subject group to form a recommendation sequence. For example, the electronic device may select images from a preset subject group.

[0343] The above-mentioned several implementation modes are only examples. In specific implementation, there may be other ways of selecting the recommendation sequence.

[0344] After the electronic device obtains multiple recommended images, it can also sort the multiple recommended images. This application does not limit the sorting method. For example, it can be sorted from high to low according to the score, sorted according to a certain theme order, sorted according to the order of storage time, etc.

[0345] S311, displaying indication information of recommended images.

[0346] The indication information of the recommended image may include a thumbnail, or may include a name, a selected selection mark box, or other indication information, which is not limited here.

[0347] In some implementations, the electronic device may launch an application that provides an image recommendation function, and then display indication information of the recommended image in a user interface provided by the application.

[0348] For example, reference Figure 2C The electronic device can display thumbnails of the multiple images traversed in S307 in the user interface 23 of the gallery application, and mark the images recommended by the electronic device by selecting a selection mark box, and also add numbers in the corresponding selection mark boxes to indicate the ranking of the corresponding images in the recommended images.

[0349] Optionally, S312, adjust the recommended image according to the user operation.

[0350] The adjustment for the recommended images may include: adjusting the order of the multiple recommended images, adding recommended images, deleting recommended images, etc.

[0351] There may be many implementation forms of the user operation for adjusting the recommended image, which are not limited here.

[0352] For example, please refer to the above Figure 2C , which exemplifies several ways to adjust the recommended images.

[0353] S313: A user operation for generating a video is detected.

[0354] The user operation for generating the video may be implemented in various forms, such as user operation on the display screen, voice commands, gestures, expressions, etc., which are not limited here.

[0355] In some implementations, S313 may also be an optional step, and the electronic device does not need to receive the user operation, and can directly generate a video using the recommended image after determining the recommended image.

[0356] S314, generating a video according to the recommended image.

[0357] In some embodiments, after the electronic device detects a user operation for generating a video, it may first ask the user to obtain the relevant permission to use the "one-click video" function. An exemplary user interface for obtaining authorization from the user can be found in Figure 2D-2F The user interface shown is not described in detail here.

[0358] The process of generating a video based on the recommended image may include the following steps:

[0359] First, the material (i.e., the recommended images) is analyzed, for example, its size, quantity, duration, theme, and other information.

[0360] Then, match the audio and video template according to the material information. Audio and video templates can be divided into video templates and audio templates, or they can be combined into one template. Among them, the video template may include transition effects, filters, image cropping methods, visualization elements (such as stickers), etc. The audio template includes audio. The audio and video template can be stored in the electronic device, or it can be obtained by the electronic device from the cloud server through the network. In some embodiments, after matching the audio and video template, the user can also adjust it to the form they want according to actual needs.

[0361] Then, the electronic device starts to generate a video based on the matched audio and video template and the recommended material. The order in which the images appear in the generated video can be consistent with the order in which the images appear in the recommended image determined by the electronic device. If the recommended image includes a dynamic photo, the dynamic photo included in the video can be presented in full length, and the same applies to the video. In other embodiments, the user can select the audio and video template by himself.

[0362] In the above process of analyzing materials, matching templates, and generating videos, the electronic device can output some prompt information to remind the user of the steps currently being performed. Examples of these prompt information can be found in Figure 2G-Figure 2I Related description.

[0363] In some implementations, after the electronic device generates a video, it can also automatically play the video so that the user can view the video effect. Figure 2K , the user interface 30 shows an effect of automatically playing a video after generating a video. Of course, the electronic device can also play the video after receiving an operation of triggering the video to be played by the user.

[0364] S315, the electronic device saves the generated video.

[0365] The electronic device can automatically save the video after generating it.

[0366] Alternatively, the electronic device may also save the video after generating the video in response to a received user operation. For example, the user operation may be, for example, Figure 2J User operation on the export control 296C in the user interface 30 is shown.

[0367] The electronic device can save the video to a storage path under an application that provides image recommendation functions (such as a gallery), and can also store it to a storage path under a "clipping" service, or store it to both paths at the same time, without limitation here.

[0368] S316: Detect a user operation for sharing a video, and share the video to a corresponding object.

[0369] After the electronic device saves the generated video, it can display the video in a user interface based on its storage path for the user to view. Figure 2M-Figure 2O A method of displaying the video by an electronic device is exemplified.

[0370] After the electronic device detects the user operation for sharing the video, it can share the video with the corresponding object. Among them, the user operation for sharing the video can be implemented in various forms, which are not limited here, for example, it can include Fig.2O The user operation on the sharing control 352 in the video may be a device of a contact, an application in an electronic device (such as a social platform, a memo, a note, etc.), a discovered nearby device, a device where a contact in an application is located, etc. The application to which the electronic device shares the video may be referred to as a first application, and the device to which the video is shared may be referred to as a first device.

[0371] It is not limited to the electronic device displaying the video first and then sharing the video. In some other implementations, the electronic device may also first display various objects, then select one or more of the objects, and then find the generated video from the gallery or other applications and share it with the one or more objects.

[0372] Above Figure 3B The image recommendation method shown can recommend high-quality, exciting, and non-similar (i.e., different content) images to users, so that users can pay attention to these recommended images. In addition, the electronic device can also generate videos using the recommended images to meet the user's needs for making high-quality, exciting, and non-repetitive videos. The electronic device can also share the generated videos to social platforms or contacts to meet the user's needs for sharing videos.

[0373] Above Figure 3B In the method shown, after judging whether the image is a duplicate frame in S305, the judgment result can be used to delete the duplicate image. The deletion can be directly performed by the electronic device, or the electronic device shows the image recommended for deletion to the user and the user decides whether to delete the duplicate image.

[0374] Above Figure 3B In the method shown, after determining the theme of the image in S305, the electronic device may also recommend images with the same theme to the user so that the user can view more images with the same theme.

[0375] The electronic device provided by the embodiments of the present application is introduced below.

[0376] Figure 7The hardware structure diagram of the electronic device 100 provided in the embodiment of the present application is shown. The electronic device 100 is used to execute the image recommendation method provided in the above method embodiment.

[0377] The electronic device 100 may include a processor 101, a memory 102, a wireless communication module 103, a mobile communication module 104, an antenna 103A, an antenna 104A, a power switch 105, a sensor module 106, a focus motor 107, a camera 108, a display screen 109, etc. Among them, the sensor module 106 may include a gyroscope sensor 106A, an acceleration sensor 106B, an ambient light sensor 106C, an image sensor 106D, a distance sensor 106E, etc. Among them, the wireless communication module 103 may include a WLAN communication module, a Bluetooth communication module, etc. The above multiple parts can transmit data through a bus.

[0378] The processor 101 may include one or more processing units, for example, the processor 101 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0379] The memory 102 may be used to store computer executable program codes, which may include instructions. The processor 101 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 102 .

[0380] The wireless communication function of the electronic device 100 can be implemented through the antenna 103A, the antenna 104A, the mobile communication module 104, the wireless communication module 103, the modem processor and the baseband processor.

[0381] Antenna 103A and antenna 104A can be used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas.

[0382] The mobile communication module 104 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 104 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 104 can receive electromagnetic waves through the antenna 104A, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 104 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 104A.

[0383] A modem processor may include a modulator and a demodulator.

[0384] The wireless communication module 103 can provide wireless communication solutions including wireless local area networks (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 103 receives electromagnetic waves via the antenna 103A, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 101. The wireless communication module 103 can also receive the signal to be sent from the processor 101, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 103A.

[0385] The electronic device 100 can realize the shooting function through the ISP, the camera 108, the video codec, the GPU, the display screen 109 and the application processor.

[0386] The ISP is used to process the data fed back by the camera 108. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 108.

[0387] The camera 108 can be used to capture still photos or dynamic photos or videos. The object generates an optical image through the lens and projects it onto the image sensor. The image sensor can convert the optical signal into an electrical signal, and then transmit the electrical signal to the ISP to convert it into a digital image signal. The ISP can output the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0388] The video codec is used to compress or decompress digital images. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can open or save pictures or videos in multiple coding formats.

[0389] The electronic device 100 can realize the display function through a GPU, a display screen 109, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 109 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 101 may include one or more GPUs, which execute program instructions to generate or change display information.

[0390] The display screen 109 is used to display images, videos, etc. The display screen 109 includes a display panel. In some embodiments, the electronic device 100 may include 1 or N display screens 109, where N is a positive integer greater than 1.

[0391] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0392] In the embodiment of the present application, the camera 108 can be used to capture images. The AP or GPU in the processor 101 can be used to analyze the current frame preview image to obtain the label of the preview image, analyze the image saved by the user to obtain the label of the image, and determine the recommended image from multiple images. The AP or GPU in the processor 101 can also be used to generate a video based on the recommended image.

[0393] The memory 102 may be used to store preview images and labels of the preview images, images saved as specified by the user and labels of the images, and videos generated based on recommended images.

[0394] The display screen 109 may be used to display various types of information for the user, such as the user interface mentioned in the above embodiment.

[0395] Information input devices such as the display screen 109, the camera 108, and the microphone may also be used to receive user operations, and the electronic device may respond to these user operations and execute corresponding functions.

[0396] The wireless communication module 103 , the mobile communication module 104 , etc. may be used to share the video generated by the electronic device based on the recommended image to other devices.

[0397] The operations performed by the various components in the electronic device 100 may be specifically referred to the relevant description of the above method embodiment, which will not be elaborated here.

[0398] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes a mobile operating system with a layered architecture as an example to exemplify the software structure of the electronic device 100.

[0399] Figure 8 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0400] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the mobile operating system is divided into four layers, from top to bottom: application layer, application framework layer / core service layer, system library and runtime, and kernel layer.

[0401] The application layer can include a series of application packages.

[0402] like Figure 8 As shown, the application package may include applications such as camera, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. The application layer may also include the application for providing image recommendation function mentioned in the above embodiment, such as gallery or other third-party applications.

[0403] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0404] like Figure 8 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0405] A window manager is used to manage window programs.

[0406] Content providers are used to store and retrieve data and make it accessible to applications.

[0407] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc.

[0408] The phone manager is used to provide the communication function of the electronic device.

[0409] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0410] The notification manager enables applications to display notification information in the status bar, which can be used to convey notification-type messages and disappear automatically after a short stay without user interaction.

[0411] Runtime can refer to all code libraries, frameworks, etc. required for the program to run.

[0412] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0413] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0414] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0415] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0416] A 2D graphics engine is a drawing engine for 2D drawings.

[0417] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0418] It should be understood that each step in the above method embodiment can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0419] The present application also provides an electronic device, which may include: a memory and a processor. The memory may be used to store a computer program; the processor may be used to call the computer program in the memory so that the electronic device executes the method executed by the electronic device side in any of the above embodiments.

[0420] The present application also provides an electronic device, which may include: a memory and a processor. The memory may be used to store a computer program; the processor may be used to call the computer program in the memory so that the electronic device executes the method executed by the electronic device side in any of the above embodiments.

[0421] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the electronic device side in any of the above embodiments.

[0422] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0423] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0424] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0425] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0426] Exemplarily, the chip system can be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0427] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method executed by the electronic device side in any of the above embodiments.

[0428] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device side in any of the above embodiments.

[0429] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0430] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)), etc.

[0431] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0432] In short, the above is only an embodiment of the technical solution of this application, and is not intended to limit the protection scope of this application. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of this application shall be included in the protection scope of this application.

Claims

1. An image recommendation method, characterized in that: The method is applied to an electronic device, and the method comprises: Displaying a preview interface, wherein the preview interface displays a preview image frame reported by the camera of the electronic device; In the process of displaying the preview interface, determining a first score according to the image quality of the j-th preview image frame, the j-th preview image frame being a preview image frame displayed on the preview interface; Determine whether the j-th preview image frame contains a subject; if so, determine a second score, a third score, and a fourth score of the j-th preview image frame according to one or more of the composition, facial beauty, and character action of the j-th preview image frame; if not, set the second score, the third score, and the fourth score of the j-th preview image frame to 0; Calculate the weighted average of the first score, the second score, the third score, and the fourth score; Determining the weighted average score as the score of the j-th preview image frame; Detecting a user operation for capturing an image, and saving a first image, wherein the first image includes one or more preview image frames displayed on the preview interface; determining a theme, a score, and a repeatability of the first image, wherein the score of the first image is determined according to the score of the first image frame, the first preview image frame includes preview image frames other than transition frames among all preview image frames of the first image, the transition frames include preview image frames whose information amount and image quality do not meet a first threshold, and the repeatability indicates whether the first image is repeated with a historically captured image; determining a recommended image from the plurality of first images based on respective scores, themes, and repetitiveness of the plurality of first images; The indication information of the recommended image is displayed.

2. The method according to claim 1, characterized in that: Determining the second score, the third score, and the fourth score of the jth preview image frame according to one or more of the composition, the facial beauty, and the character action of the jth preview image frame specifically includes: determining a second score of the j-th preview image frame according to the composition; In the case that the j-th preview image frame contains a subject, determining a third score of the j-th preview image frame according to the subject category of the j-th preview image frame; When the subject included in the j-th preview image frame is a person, a fourth score of the j-th preview image frame is determined according to an action of the person.

3. The method according to claim 1, characterized in that: Determining a first score according to the image quality of the j-th preview image frame specifically includes: A first score is determined according to the clarity and / or symmetry of the j-th preview image frame; the higher the clarity and / or symmetry, the higher the first score.

4. The method according to claim 2, characterized in that: Determining a third score of the jth preview image frame according to the subject category of the jth preview image frame specifically includes: In the case where the subject of the j-th preview image frame includes an animal, if the j-th preview image frame includes the face of the animal, the third score is set to a high score, wherein the high score includes a score whose difference from the full score is less than a preset value, and the full score is the highest score in the scoring range of the third score; or, In the case that the subject of the j-th preview image frame includes a person, the third score is determined based on any one or more of whether the j-th preview image frame includes a face, whether the eyes of the included face are open or closed, the expression of the face, the posture of the face, the area of ​​the face in the j-th preview image frame, the area size, the proportion, and the number of faces.

5. The method according to claim 2 or 4, characterized in that: In the case where the subject of the j-th preview image frame includes a person, determining the third score specifically includes: The third score is determined according to the following formula: is the third score; N face is the number of faces contained in the j-th preview image frame; s e，i is the eye-opening and eye-closing score of the i-th face. When the i-th face opens its eyes, s e，i is set to the first value, when the i-th face closes its eyes, s e，i is set to a second value, the first value being higher than the second value; s m，i is the smile score of the i-th face. When the i-th face smiles, s m，i is set to the third value, when the i-th face is not smiling, s m，i is set to a fourth value, the third value being higher than the fourth value; s d，i is the posture score of the i-th face. When the i-th face faces the camera, s d，i is set to the fifth value, when the i-th face is not facing the camera, s d，i is set to a sixth value, the fifth value being higher than the sixth value; s b is the face size influence coefficient of the i-th face. When the size of the i-th face is within the first range, s b is set to the seventh value, when the size of the i-th face is outside the first range, s b is set to an eighth value, the seventh value being greater than the eighth value; s N is the face proportion coefficient of the i-th face. The larger the area proportion of the i-th face, the larger the N The bigger; s c is the composition beauty coefficient. When the i-th face is located in the first area of ​​the j-th preview image frame, s c is set to the ninth value, when the i-th face is located outside the first area of ​​the j-th preview image frame, s c is set to a tenth value, the ninth value is higher than the tenth value, and the first area includes a central area of ​​the j-th preview image frame; s p is the face-free penalty threshold. When the j-th preview image frame contains a face, s p is 0, and the jth preview image frame does not contain a face, s p Set to the preset faceless penalty threshold.

6. The method according to claim 5, characterized in that The first value, the third value, and the fifth value are all high scores; The second value, the fourth value, and the sixth value are all low scores; The high score includes a score whose difference with the full score is less than a preset value, the low score includes a score whose difference with the full score is greater than a preset value, and the full score is the highest score in the scoring range of the third score.

7. The method according to claim 5 or 6, characterized in that: When the distance between the i-th face and the electronic device is different, the same size of the i-th face corresponds to different s b .

8. The method according to claim 2, characterized in that: Determining a fourth score of the jth preview image frame according to the action of the character specifically includes: Determining whether the person in the j-th preview image frame includes a preset wonderful action; If so, the fourth score is equal to the product of the confidence level output by the action evaluation output algorithm and the high score; If not, setting the fourth score to a low score; Among them, the high score includes a score whose difference with the full score is less than a preset value, the low score includes a score whose difference with the full score is greater than a preset value, and the full score is the highest score in the scoring range of the fourth score.

9. The method according to claim 1, characterized in that: Determining a score of the first image specifically includes: Determine the wonderful frame with the highest score in the first preview image frame; Determine a weighted average score of the wonderful frame, a first preset number of preview image frames before the wonderful frame in the first preview image frame, and a second preset number of preview image frames after the wonderful frame in the first preview image frame; The weighted average score is determined as the score for the first image.

10. The method according to any one of claims 1 to 9, characterized in that: Determining the repeatability of the first image specifically includes: The repeatability of the first image is determined according to any one or more of the scores of the first image and the historically shot image, including whether the person is moving, the subject, and the similarity.

11. The method according to any one of claims 1 to 10, characterized in that Determining the repeatability of the first image specifically includes: If any one of the first image and the historically shot images is repeated, it is determined that the first image is repeated; if the first image and all of the historically shot images are not repeated, it is determined that the first image is not repeated.

12. The method according to claim 10 or 11, characterized in that: The method further comprises: Only in one of the following cases, the first image and the second image in the historically shot images are not repeated, and the second image is any one of the historically shot images: The score of the first image is higher than the score of the second image; Alternatively, the score of the first image is lower than the score of the second image, and one of the first image and the second image is a motion frame, and the other is not a motion frame; Alternatively, the score of the first image is lower than the score of the second image, and both the first image and the second image are motion frames, and the motion time is greater than a preset minimum time interval; Alternatively, the score of the first image is lower than the score of the second image, and both the first image and the second image are non-motion frames, and the subjects are the same, and the similarity between the first image and the second image is lower than a second threshold.

13. The method according to any one of claims 1 to 12, characterized in that: The theme of the first image includes one of the following: children, animals, buildings, plants, and landscapes; the theme of the first image is determined according to the content of the first image.

14. The method according to any one of claims 1 to 13, characterized in that: Determining a recommended image from the plurality of first images according to respective scores, themes, and repetitiveness of the plurality of first images specifically includes: Dividing the plurality of first images into a plurality of subject groups according to respective subjects of the plurality of first images; For each of the subject groups, the first images are sorted according to the scores and repetitiveness, with the higher scores being arranged first, and the non-repetitive images being arranged first compared to the repetitive images; The first images are screened from each subject group according to the order of precedence to form recommended images.

15. The method according to any one of claims 1 to 14, characterized in that: Before determining a recommended image from the plurality of first images based on the scores, themes, and repetitiveness of the plurality of first images, the method further includes: Displaying a notification message, wherein the notification message is used to prompt the user to view the recommended image; A user operation acting on the notification message is detected.

16. The method according to claim 15, characterized in that The notification message is displayed at any of the following times: Preset fixed time; After a third preset number of images have been captured; After entering a first location, taking a plurality of images at the first location, and after leaving the first location.

17. The method according to any one of claims 1 to 16, characterized in that: The instruction information for displaying the recommended image specifically includes: Thumbnails of the recommended images are displayed.

18. The method according to any one of claims 1 to 17, characterized in that: The instruction information for displaying the recommended image specifically includes: The indication information of the plurality of first images is displayed, and the indication information of the recommended image is selected.

19. The method according to any one of claims 1 to 18, characterized in that: After displaying the indication information of the recommended image, the method further includes: A user action for generating a video is detected; A video is generated according to the recommended image.

20. The method according to claim 19, characterized in that Generating a video according to the recommended image specifically includes: Matching audio and video templates according to the recommended image; A video is generated according to the recommended image and the matching audio and video template.

21. The method according to claim 19 or 20, characterized in that After generating a video according to the recommended image, the method further includes: A user operation for sharing a video is detected; The video is shared to a first application in the electronic device, or the video is shared to a first device different from the electronic device.

22. The method according to any one of claims 1 to 21, characterized in that: When the first image is a dynamic photo, saving the first image specifically includes: The moment when the user operation for capturing an image is detected, the moment before and the moment after the moment, the preview image frame displayed on the preview interface, and the sound detected by the electronic device are stored as dynamic photos.

23. An electronic device, characterized in that: include: A memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1-22.

24. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device executes the method as described in any one of claims 1 to 22.

25. A computer program product, characterized in that When the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1 to 22.

Images