Image recommendation method and related apparatus
By detecting the subject, score, and repetition of images in electronic devices, and scoring them based on factors such as image quality, composition, facial aesthetics, and human movement, high-quality, non-repeating images are recommended. This solves the problem of users having difficulty finding the images they need and enables efficient image recommendation and video generation and sharing.
Patent Information
- Application Number
- PCT/CN2024/081113
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-03-12
- Publication Date
- 2024-11-21
AI Technical Summary
Users often struggle to find the images they need after taking numerous photos and videos. Existing image recommendation solutions for smart devices are inefficient and fail to effectively improve the user experience.
By detecting the subject, score, and repetition of images in electronic devices, and scoring them based on factors such as image quality, composition, facial aesthetics, and human movement, high-quality, unique images are recommended and generated into videos or shared on social media platforms.
It improves the efficiency and accuracy of image recommendations, saves users time, meets users' needs for high-quality images, and supports one-click generation and sharing of videos.
Smart Images

Figure CN2024081113_21112024_PF_FP_ABST
Abstract
Description
Image recommendation method and related device
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 17, 2023, with application number 202310558291.5 and application name “Image Recommendation Method and Related Device”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the fields of terminals and image analysis, and in particular to image recommendation methods and related devices. Background Art
[0003] Smart devices, such as mobile phones, are equipped with camera functions, allowing users to capture desired images, such as photos, animated photos, and videos. However, after capturing a large number of photos and videos, it can be difficult to accurately find the photo or video they are looking for. Therefore, some smart devices offer image recommendation solutions, which can recommend images for users to view, saving time and providing convenience. To enhance the user experience, how smart devices recommend images is a current and future research direction.
[0004] Summary of the Invention
[0005] This application provides an image recommendation method and related devices that can recommend images to users.
[0006] In a first aspect, a method for recommending images is provided for use in an electronic device. The method may include: displaying a preview interface displaying preview image frames reported by a camera of the electronic device; detecting a user operation for capturing an image, saving a first image, the first image comprising one or more preview image frames displayed in the preview interface; determining a theme, score, and repetitiveness of the first image, wherein the score of the first image is derived based on any one or more of the first image's image quality, composition, facial aesthetics, and character movement, and the repetitiveness indicates whether the first image is a duplicate of a historically captured image; determining a recommended image from a plurality of first images based on their respective scores, themes, and repetitiveness; and displaying information indicating the recommended image.
[0007] By implementing the method of the first aspect, the electronic device can recommend images to the user based on the subject, score, and repetitiveness of each image among multiple captured images.
[0008] In conjunction with the first aspect, in some embodiments, the method may further include: during display of the preview interface, determining a score for each preview image frame displayed in the preview interface, wherein the score of the preview image frame is derived based on any one or more of image quality, composition, facial aesthetics, and character motion of the preview image frame. Determining the score of the first image specifically includes: determining the score of the first image based on the scores of one or more preview image frames contained in the first image.
[0009] Based on the previous implementation, determining the score of the preview image frame in real time during the preview process can save the execution time of the subsequent determination of the first image score, reducing time costs. In addition, since the preview stream has a smaller storage capacity, this also facilitates the calculation and analysis process.
[0010] In combination with the previous embodiment, the score of each preview image frame displayed on the preview interface can be determined in the following manner: a first score is determined based on the image quality of the jth preview image frame, where the jth preview image frame is any preview image frame displayed on the preview interface; whether the jth preview image frame contains a subject is determined, and if so, a second score of the jth preview image frame is determined based on the composition; if not, the second score, third score, and fourth score of the jth preview image frame are all set to 0; when the jth preview image frame contains a subject, the third score of the jth preview image frame is determined based on the subject category of the jth preview image frame; when the subject contained in the jth preview image frame is a person, the fourth score of the jth preview image frame is determined based on the action of the person; a weighted average score of the first score, the second score, the third score, and the fourth score is calculated; and the weighted average score is determined as the score of the jth preview image frame.
[0011] In this way, the score of each preview image frame can reflect the image quality, composition, facial beauty and character movement of the preview image frame, so that the score of the final first image can also reflect the above factors.
[0012] Optionally, the first score, the second score, the third score, and the fourth score are all within a first score range, wherein the first score range depends on a preset scoring system, such as a 10-point system or a 100-point system.
[0013] Optionally, the electronic device may determine the first score based on the clarity and / or symmetry of the j-th preview image frame; the higher the clarity and / or symmetry, the higher the first score.
[0014] Optionally, the electronic device may determine the third score of the j-th preview image frame according to the following method:
[0015] In the case where the subject of the j-th preview image frame includes an animal, if the j-th preview image frame includes the face of the animal, the third score is set to a high score, where the high score includes a score whose difference from the full score is less than a preset value, and the full score is the highest score in the scoring range of the third score;
[0016] or,
[0017] When the subject of the jth preview image frame includes a person, the third score is determined based on whether the jth preview image frame contains a human face, whether the eyes of the contained human face are open or closed, the expression of the face, the posture of the face, the area of the face in the jth preview image frame, the area size, the proportion, and the number of faces.
[0018] For example, for animals, as long as the face is captured, the user considers the image content acceptable, and the facial beauty score can be given a high one. However, users have higher expectations for the beauty of human faces, so the facial beauty score for tasks can be considered based on factors such as the person's expression, facial posture, the face's proportion in the image, and the number of faces. This scoring method better meets user expectations.
[0019] Optionally, when the subject of the j-th preview image frame includes a person, the electronic device may determine the third score by the following formula:
[0020] For the third score;
[0021] N face is the number of faces contained in the j-th preview image frame;
[0022] s e,i is the score of the eyes open and closed of the i-th face. When the i-th face opens its eyes, s e,i Set to the first value, when the i-th face closes its eyes, s e,i is set to a second value, the first value being higher than the second value;
[0023] s m,i is the smile score of the i-th face. When the i-th face smiles, s m,i Set to the third value, when the i-th face is not smiling, s m,i Set to the fourth value, the third value is higher than the fourth value;
[0024] s d,i is the posture score of the i-th face. When the i-th face faces the camera, s d,i Set to the fifth value, when the i-th face is not facing the camera, s d,i Set to the sixth value, the fifth value is higher than the sixth value;
[0025] sb is the face size influence coefficient of the i-th face. When the size of the i-th face is within the first range, s b Set to the seventh value, when the size of the i-th face is outside the first range, s b Set to the eighth value, the seventh value is greater than the eighth value;
[0026] s N is the face ratio coefficient of the i-th face. The larger the area ratio of the i-th face, the higher the s N The bigger;
[0027] s c is the composition beauty coefficient. When the i-th face is located in the first area of the j-th preview image frame, s c Set to the ninth value, when the i-th face is outside the first area of the j-th preview image frame, s c is set to the tenth value, the ninth value is higher than the tenth value, and the first area includes the central area of the j-th preview image frame;
[0028] s p is the face-free penalty threshold. When the jth preview image frame contains a face, s p When s is 0, the jth preview image frame does not contain a face. p Set to the preset faceless penalty threshold.
[0029] The above formula provides a specific way to determine the score for measuring the beauty of a person's face.
[0030] Optionally, in the above formula, the first value, the third value, and the fifth value are all high scores; the second value, the fourth value, and the sixth value are all low scores; among them, the high score includes the score whose difference from the full score is less than the preset value, the low score includes the score whose difference from the full score is greater than the preset value, and the full score is the highest score in the scoring range of the third score.
[0031] Optionally, in the above formula, when the distance between the i-th face and the electronic device is different, the same i-th face size corresponds to different s b This allows us to comprehensively evaluate the beauty of faces at different distances, limiting the scores of faces that are too large or too small.
[0032] Optionally, the electronic device may determine the fourth score of the jth preview image frame in the following manner: determine whether the character in the jth preview image frame contains a preset wonderful action; if so, the fourth score is equal to the product of the confidence level output by the action evaluation output algorithm and the high score; if not, the fourth score is set to a low score; wherein, the high score includes a score whose difference from the full score is less than a preset value, the low score includes a score whose difference from the full score is greater than a preset value, and the full score is the highest score in the scoring range of the fourth score.
[0033] In combination with the previous embodiment, in some embodiments, the score of the first image is determined based on the scores of the preview image frames contained in the first image, specifically including: determining the score of the first image based on the first preview image frame, the first preview image frame including preview image frames other than transition frames among all preview image frames of the first image, and the transition frames including preview image frames whose information amount and image quality do not meet the first threshold.
[0034] By removing the transition frame, the score of the first image can more objectively reflect the quality of the first image.
[0035] In combination with the previous embodiment, the electronic device can determine the score of the first image based on the first preview image frame in the following manner: determine the wonderful frame with the highest score in the first preview image frame; determine the weighted average score of the wonderful frame, the first preset number of preview image frames before the wonderful frame in the first preview image frame, and the second preset number of preview image frames after the wonderful frame in the first preview image frame; and determine the weighted average score as the score of the first image.
[0036] Optionally, the first preset number and the second preset number may be the same or different.
[0037] Optionally, the weighted coefficient of the wonderful frame is 1, and the formaldehyde coefficients of other image frames preceding or following the wonderful frame decrease in sequence.
[0038] In combination with the first aspect, in some embodiments, the electronic device may determine the repeatability of the first image based on a score between the first image and historically captured images, including whether a person is moving, a subject, and any one or more of similarities.
[0039] In combination with the first aspect, in some embodiments, if any one of the first image and the historically captured images is repeated, the first image is determined to be repeated; if the first image and all of the historically captured images are not repeated, the first image is determined to be non-repeated.
[0040] In combination with any one of the above two embodiments, only in any one of the following cases, the first image and the second image in the historical images are not repeated, and the second image is any one of the historical images:
[0041] The score of the first image is higher than the score of the second image;
[0042] Alternatively, the score of the first image is lower than the score of the second image, and one of the first image and the second image is a motion frame, and the other is not a motion frame;
[0043] Alternatively, the score of the first image is lower than the score of the second image, and both the first image and the second image are motion frames, and the motion time is greater than a preset minimum time interval;
[0044] Alternatively, the score of the first image is lower than the score of the second image, and both the first image and the second image are non-motion frames, and the subjects are the same, and the similarity between the first image and the second image is lower than a second threshold.
[0045] In combination with the first aspect, in some embodiments, the subject of the first image includes any one of the following: children, animals, buildings, plants, and scenery; and the subject of the first image is determined according to the content of the first image.
[0046] In conjunction with the first aspect, in some embodiments, the electronic device may determine recommended images from multiple first images in the following manner: categorize the multiple first images into multiple theme groups based on their respective themes; sort the first images in each theme group by their scores and repetition rate, with images with higher scores ranked first and images with no repetition ranked first compared to images with higher repetition rates; and filter the first images in each theme group based on their ranking order to generate recommended images. This allows for the recommendation of high-quality, thematically diverse, and non-repetitive images to the user.
[0047] A certain number of recommended images are selected from each theme group to ensure the diversity and high quality of the final recommended images.
[0048] Optionally, the number of recommended images selected by the electronic device from each theme group may be the same or different.
[0049] In combination with the first aspect, in some embodiments, before determining a recommended image from a plurality of first images based on their respective scores, themes, and repeatability, the method may further include: displaying a notification message, the notification message being used to prompt the user to view the recommended image; and detecting a user operation acting on the notification message.
[0050] In combination with the previous embodiment, the notification message is displayed at any of the following time points: a preset fixed time; after the third preset number of images are taken; after entering the first location and taking multiple images at the first location, and after leaving the first location.
[0051] In conjunction with the first aspect, in some embodiments, the manner of displaying the indication information of the recommended image may include the following two:
[0052] 1. Display thumbnails of recommended images.
[0053] 2. Displaying indication information of a plurality of first images, wherein indication information of a recommended image is selected.
[0054] In conjunction with the first aspect, in some embodiments, after displaying the recommended image indication information, the method may further include: detecting a user operation for generating a video; and generating a video based on the recommended image. This facilitates one-click video generation, meeting the user's video creation needs.
[0055] In combination with the previous embodiment, the electronic device may generate a video based on the recommended image in the following manner: matching an audio and video template based on the recommended image; and generating a video based on the recommended image and the matched audio and video template.
[0056] In conjunction with the previous embodiment, after generating a video based on a recommended image, the method may further include: detecting a user operation for sharing the video; and sharing the video to a first application on the electronic device, or sharing the video to a first device other than the electronic device. This allows users to quickly and easily share videos created using recommended images to various applications or devices, satisfying their video sharing needs.
[0057] In conjunction with the first aspect, in some embodiments, the type of any first image is any of the following: a still photo comprising a single image frame; a dynamic photo comprising multiple image frames; or a video comprising multiple image frames. That is, the image recommendation method provided herein can recommend different types of images to a user.
[0058] In combination with the first aspect, in some embodiments, when the first image is a dynamic photo, saving the first image specifically includes: storing the moment when the user operation for taking the image is detected, before the moment, and after the moment, the preview image frame displayed on the preview interface, and the sound detected by the electronic device as a dynamic photo.
[0059] In combination with the first aspect, in some embodiments, the historical captured images specifically include any one of the following: all images taken by the electronic device before the first image; images taken after the camera was last started; images taken on the same day; images within a fourth preset number taken recently; images taken at the first location after entering the first location.
[0060] In combination with the first aspect, in some embodiments, the multiple first images specifically include any one of the following: all images taken by the electronic device; images taken between the last time the camera was started and the last time the camera was turned off; images taken on the same day; the fifth preset number of images taken most recently; images taken at the first location between entering the second location and leaving the second location.
[0061] In a second aspect, an electronic device is provided, comprising: a memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute a method executed by the electronic device in the first aspect or any one of the embodiments of the first aspect.
[0062] In a third aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, enables the electronic device to execute a method as executed by the electronic device in the first aspect or any one of the embodiments of the first aspect.
[0063] In a fourth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method executed by the electronic device in the first aspect or any one of the embodiments of the first aspect.
[0064] In a fifth aspect, an embodiment of the present application provides a chip system, which includes at least one processor for implementing the method executed by the electronic device in the first aspect or any one embodiment of the first aspect.
[0065] The image recommendation method provided in this application is implemented to provide an intelligent image recommendation solution that can recommend images to users based on the theme, score, and repetitiveness of each image to meet the actual needs of users. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Figures 1A-1C are user interfaces for taking dynamic photos according to an embodiment of the present application;
[0067] Figures 2A-2P are user interfaces for recommending images, generating videos, and sharing videos, provided in an embodiment of the present application;
[0068] FIG3A is a flowchart of an image recommendation method provided in an embodiment of the present application;
[0069] FIG3B is a flowchart of another image recommendation method provided by an embodiment of the present application;
[0070] FIG4 is a flowchart of scoring a preview image of a current frame provided by an embodiment of the present application;
[0071] FIG5 is a diagram of the face size influence coefficient s provided in an embodiment of the present application. b The mapping relationship curve between the face size and the face size;
[0072] FIG6 is a flow chart of determining the repeatability of two images according to an embodiment of the present application;
[0073] FIG7 is a block diagram of the hardware structure of an electronic device provided in an embodiment of the present application;
[0074] FIG8 is a software architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0075] The following is a clear and detailed description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.
[0076] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.
[0077] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.
[0078] This application provides an image recommendation method and related devices.
[0079] The image recommendation method is applied to electronic devices, and the electronic devices are smart terminal devices, which can be of various types. The embodiments of the present application do not limit their specific types. For example, the electronic device can be a mobile phone, and can also include a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, a smart screen, a wearable device, an augmented reality (AR) device, a virtual reality (VR) device, an artificial intelligence (AI) device, a car machine, a smart headset, a game console, and can also be an Internet of Things (IOT) device or a smart home device such as a smart TV, etc. Without limitation, the electronic device can also include a laptop computer with a touch-sensitive surface or touch panel, a desktop computer with a touch-sensitive surface or touch panel, and other non-portable terminal devices, etc.
[0080] In this image recommendation method, the electronic device can score multiple images separately and determine whether each image is similar to another. The score of an image measures the quality of the image and the degree of content, and the similarity indicates whether an image is similar to other images. Based on the scores and similarities of each image, the electronic device can recommend high-quality and content-rich, non-similar (i.e., different content) images to the user, so that the user can pay attention to these recommended images. For the scoring strategy and the similarity measurement method, please refer to the detailed description of the method embodiment below, which will not be repeated here.
[0081] In some embodiments, the electronic device may perform further processing based on these recommended images. For example, the electronic device may generate a video using its recommended images, meeting the user's need to create high-quality videos with engaging content and no repetitive content. This function of generating videos using recommended images may be called "one-click video creation" or another name. For another example, the electronic device may also share the generated video on social platforms or with contacts, meeting the user's need to share videos.
[0082] The image recommendation method provided in this application can be a function provided by the system's gallery application or a function provided by other third-party applications, which is not limited here. System applications refer to applications provided by the manufacturer of the electronic device, and third-party applications are applications not provided by the device manufacturer.
[0083] Among them, the gallery is an image management APP on electronic devices such as smartphones and tablets, which can also be called "album" or "photo", etc. This application does not limit the name of the application. The gallery can support users to perform various operations on images stored on electronic devices, such as browsing, editing, deleting, selecting, etc. That is, the object managed by the "gallery" is the image. In other cases, the gallery can also support users to perform the above-mentioned operations on images stored on cloud servers. The gallery APP in the following embodiments of this application also provides an image recommendation function.
[0084] For the sake of simplicity, the following description will use the image recommendation method of the present application provided by the image gallery as an example.
[0085] The images mentioned in this application may include the following categories:
[0086] 1. Live Photo. A live photo includes the moment of capture, along with the image content and sound from the time before and after that moment. This means it contains multiple frames (e.g., 90 frames) and the corresponding sound, and can be played like a short video. The number of frames before and after the moment of capture can be the same or different. Live photos are also called animated images, live photos, etc.
[0087] 2. Still photos. A still photo consists of a single frame. It can be the image captured at the moment of capture, or a composite of several frames before and after the capture. A still photo contains only the image and not the sound. Both still and moving photos can be collectively referred to as photos.
[0088] 3. Video. A video can include visual content and sound recorded continuously over a period of time. Videos are generally longer than moving photos.
[0089] The electronic device can display a thumbnail of an image. The original image can be stored locally on the electronic device or in the cloud. The thumbnail of a dynamic photo can be a frame captured at the time of the photo.
[0090] The image displayed on the electronic device may be captured by the electronic device using a camera, obtained from other applications or downloaded from the Internet, or shared by other electronic devices, without limitation here.
[0091] Before introducing the image recommendation method provided by this application in detail, a set of UI examples on electronic devices are provided to help understand the method. In this set of UIs, dynamic photos are used as an example to introduce the image type.
[0092] 1A-1C are user interfaces for an electronic device to take dynamic photos.
[0093] FIG. 1A exemplarily shows an exemplary user interface 11 on an electronic device 100 for displaying installed applications.
[0094] The user interface 11 may include a status bar, a page indicator, a tray with common application icons, other application icons, etc. The user interface 11 may also include a navigation bar, a sidebar, etc. The user interface 11 shown in FIG1A may be referred to as a home screen.
[0095] The application icons in the user interface 11 may include, for example, a camera icon 111, a gallery icon 112, and icons of other applications. A camera is an application on an electronic device that uses the camera to capture images. This embodiment does not limit the name of this application. The camera application may be a system application provided by the electronic device or a third-party application.
[0096] As shown in FIG1A , the electronic device may receive a user operation (e.g., a click operation, a touch operation) on a camera icon 111, and then, in response to the operation, launch a camera application and display a user interface 12 as shown in FIG1B . The user interface 12 in FIG1B is a preview interface provided by the camera application.
[0097] As shown in FIG1B , the user interface 12 may include a mode bar 121 , a photo control 122 , a preview window 123 , a review control 124 , and a shortcut function area 125 . Specifically:
[0098] The mode bar 121 may include multiple photo mode options, such as "night scene", "portrait", "photo", "video recording", etc. Different photo modes can provide users with photo services with different effects. Users can select any of the multiple photo modes to take photos according to different needs. For example, "photo recording" may be the default photo mode, used for taking photos. "Video recording" is used to record videos. The "night scene" mode is suitable for photo scenes with low light, such as at night. The "portrait" mode is suitable for photo scenes where the subject is a person. The electronic device can detect the user operation of switching modes, such as swiping left / right in the mode bar 121, and change the currently used photo mode according to the operation. For example, when it is detected that the mode bar 121 is dragged to the left (left swiping operation) and the float stops at the "portrait" option, the electronic device can switch to the "portrait" mode. By default, the electronic device first uses the "photo recording" mode.
[0099] The photo control 122 is used to trigger a photo. The electronic device can detect whether a user operation, such as a click, is applied to the photo control 122. Upon detecting a user operation on the photo control 122, the electronic device can generate a photo instruction. Based on the photo instruction, the electronic device can obtain the image reported by the corresponding camera and save it as a still photo or a moving photo.
[0100] The preview window 123 can be used to display images captured by the electronic device's camera in real time. In different shooting modes, the electronic device can process the images reported by the camera to improve the image display effect. For example, in "portrait" mode, the electronic device can blur the background in the image reported by the camera to highlight the portrait. Here, the preview window 123 can display the image processed by the image processing algorithm corresponding to the different shooting modes in real time, so that the user can perceive the shooting effect corresponding to the different shooting modes in real time. The image in the preview window 123 can also be called a preview image.
[0101] The review control 124 can be used to display a thumbnail of a recently taken photo or video. When a user operation acting on the review control 124 is detected, the electronic device can also display an image corresponding to the thumbnail.
[0102] The quick function area 125 may include a high-dynamic range image (HDR) function option, a dynamic photo function option 125A, an AI function option, a flash function option, a color mode function option, a setting function option, and the like. The HDR function option can be used to trigger the electronic device to use the HDR algorithm to fuse the image when it is turned on. The AI function option can be used to trigger the electronic device to recognize the photo scene in the preview screen when it is turned on. The current AI function option is in the off state. The flash control can be used to trigger the electronic device to turn on or off the flash. The color mode function option can be used to trigger the electronic device to use a color filter to process the image captured by the camera. The setting function option can be used to set the photo parameters of the electronic device (for example, image size, image storage format, etc.).
[0103] The dynamic photo function option 125A can be used to turn on or off the dynamic photo shooting mode. The display state of the dynamic photo function option 125A in FIG1B indicates that the dynamic photo shooting mode has been turned on.
[0104] After turning on the dynamic photo shooting mode, the electronic device can detect user operations (such as click operations, touch operations, etc.) acting on the photo control 122, and then respond to the user operation to save the preview image and the collected sound at the photo moment and a period of time before and after the photo moment as a dynamic photo.
[0105] As shown in FIG1C , after taking a dynamic photo, a thumbnail of the dynamic photo will be displayed in the review control 124. If the user clicks the review control 124 at this time, the electronic device will start the gallery and jump to the user interface showing the dynamic photo corresponding to the thumbnail in the gallery.
[0106] Following the same operations as shown in FIG1B-FIG1C, the user can change the angle of the electronic device, or change the camera, etc., to take more dynamic photos when the framing is appropriate.
[0107] FIG2A is a user interface 21 provided by a gallery for displaying multiple images taken by a user.
[0108] As shown in Figure 2A, user interface 21 displays thumbnails of multiple dynamic photos taken by the user, including thumbnails 211A through 211P. A dynamic photo indicator is displayed in the lower left corner of each thumbnail, indicating that the image corresponding to the thumbnail is a dynamic photo. Figure 2A may be the user interface displayed after the electronic device detects a user operation on the gallery icon 112 in Figure 1A.
[0109] 2B-2L are user interfaces for recommending images from the gallery and generating videos.
[0110] The user interface 22 shown in Figure 2B is provided by the drop-down notification bar of the electronic device. The notification bar is a system application that provides notification messages in the electronic device. The user interface 22 displays a notification message 221 provided by the notification bar, and the specific content of the notification message 221 comes from the gallery. The notification message 221 includes: the gallery icon 221A, the name of the gallery 221B, the generation time 221C of the notification message, and the notification content 221D. The notification content 221D can prompt the user that the electronic device recommends images, and the user can use these images to generate videos. The notification content 221D can be, for example, "One-click blockbuster recommendation" or "Recommend you to create a wonderful short story about nature with one click."
[0111] The generation time of the notification message 221 will be described in detail in the subsequent method embodiments.
[0112] As shown in FIG2B , after the electronic device detects a user operation on the notification message 221 , it may jump to the user interface 23 displaying the gallery shown in FIG2C .
[0113] As shown in Figure 2C, the user interface 23 displays: a return key 231, a title bar 232, thumbnails of multiple images taken by the electronic device over a period of time such as thumbnails 211A to thumbnails 211P, an edit bar 233, a clear control 234, and a control 235 for generating a video.
[0114] The return key 231 is used to monitor user operations. After the electronic device detects the user operation acting on the return key 231, it can respond to the user operation and display the homepage provided by the gallery for displaying images, or return to the desktop.
[0115] Title bar 232 indicates the images currently displayed in user interface 23. For example, "All Photos" in FIG2C refers to photos taken by the electronic device over a period of time. A drop-down arrow is also provided next to title bar 232. The user can click on the drop-down arrow to select the image type and image range (including time range) they wish to view from the drop-down options in the electronic device.
[0116] The images corresponding to thumbnails 211A to 211P are multiple images captured by the electronic device within a period of time. This period of time can be defined in various ways. For example, these images can include images captured between the last time the electronic device activated the camera and the last time it was turned off, or images captured by the electronic device within the same day, or a preset number of images captured recently by the electronic device, or images captured between the time the electronic device entered a certain location / area and the time it left that location / area, and so on.
[0117] A zoom icon may be displayed in the upper left corner of each thumbnail. The user may click on the zoom icon, and the electronic device may then display the original image of the corresponding thumbnail in full screen for user convenience.
[0118] The image category identifier can also be displayed in the lower left corner of each thumbnail. For example, the images corresponding to the thumbnails in Figure 2C are all dynamic photos, so the dynamic photo identifier is displayed in the lower left corner. If the image is a video, the video identifier can be displayed in the lower left corner; if the image is a still photo, the identifier may not be displayed in the lower left corner.
[0119] A selection mark box can also be displayed in the lower right corner of each thumbnail. As shown in Figure 2C, the selection mark boxes of some thumbnails in the multiple thumbnails are empty, indicating that the corresponding thumbnails are not selected; the backgrounds of the selection mark boxes of some thumbnails (such as thumbnail 211D, thumbnail 211E, thumbnail 211G, thumbnail 211J, thumbnail 211K, and thumbnail 211M) are filled, indicating that the corresponding thumbnails are selected by the electronic device. The selection mark boxes of the selected thumbnails are also marked with numbers, and the numbers from small to large correspond to the order in which the images are selected. For example, the order of selection from front to back is: thumbnail 211M, thumbnail 211K, thumbnail 211J, thumbnail 211G, thumbnail 211E, and thumbnail 211D. The original image corresponding to the selected thumbnail is the image recommended by the electronic device to the user. For specific recommendation strategies, please refer to the introduction of the method embodiment below.
[0120] The electronic device may detect a user operation acting on a blank selection mark box and mark the corresponding thumbnail as being selected.
[0121] The edit bar 233 displays some of the thumbnails selected by the electronic device. The electronic device can detect left / right swipes on the edit bar 233 and, in response, switch to displaying other thumbnails selected by the electronic device. Each thumbnail in the edit bar 233 may also display an image category indicator in the lower left corner.
[0122] The edit bar 233 can be used to edit the thumbnails selected by the electronic device. Such editing operations include, for example, deleting, adjusting the order, and the like. For example, a delete control is displayed in the upper right corner of each thumbnail in the edit bar 233. The electronic device can detect a user operation on the delete control of a thumbnail, and then, in response to the user operation, mark the corresponding thumbnail as unselected and remove it from the edit bar 233. For another example, the electronic device can also detect a user operation of long-pressing a thumbnail in the edit bar 233 and then dragging it to another location in the edit bar 233. In response to the user operation, the electronic device can adjust the order in which the corresponding thumbnails are selected, such as moving them forward or backward.
[0123] The clear control 234 can be used to monitor user operations. After the electronic device detects the user operation acting on the clear control 234, it can respond to the user operation and mark all previously selected thumbnails as unselected, thereby achieving the purpose of clearing the selection.
[0124] The number of thumbnails currently selected by the electronic device may be displayed on the control 235 , and the number may change as the user adds or deletes selected thumbnails.
[0125] The control 235 can be used to monitor user operations. After the electronic device detects the user operation acting on the control 235, it can respond to the user operation and synthesize the original image corresponding to the selected thumbnail into a video.
[0126] If the electronic device is enabling the "one-click to create a picture" function for the first time, the electronic device may also display the user interface 24 shown in FIG2D after detecting the user operation on the control 235. The user interface 24 is used to obtain authorization for the relevant permissions required for the "one-click to create a picture" function.
[0127] As shown in FIG. 2D , the user interface 24 displays: prompt information 241 , prompt information 242 , a cancel control 243 , and an agree control 244 .
[0128] Prompt information 241 is used to inform the user of the service or application that will be launched after turning on "One-click Video Generation." This is also the service or application that supports the implementation of "One-click Video Generation." For example, the "Edit" service mentioned in Figure 2D provides functions such as video capture, adjustment, and cropping. The "Crop" service can also be called other names, which are not limited here. The "Crop" service can be a system service or a third-party service.
[0129] Prompt message 242 is used to prompt the user to obtain the user permissions required to start the "Clip" service. For example, Figure 2D shows that network permissions are required to use the "Clip" service. Prompt message 242 can also provide some controls for the user to click to view more detailed instructions.
[0130] The cancel control 243 is used to monitor user operations. After the electronic device detects the user operation acting on the cancel control 243, it can respond to the user operation and return to displaying the user interface 23 shown in Figure 2C.
[0131] The consent control 244 is used to monitor user operations. After the electronic device detects the user operation acting on the consent control 244, it can respond to the user operation. The electronic device can obtain user authorization to use the "Clip" service, and thus can use the "Clip" service via the network.
[0132] After the electronic device detects the user operation acting on the consent control 244, it can successively display the user interface 25 shown in Figure 2E and the user interface 26 shown in Figure 2F.
[0133] As shown in FIG2E , the user interface 25 displays a prompt message prompting the user to grant the "Clip" service access to music and audio in the electronic device, as well as a prohibition authorization control and an always allow control. After the electronic device detects that the always allow control is activated, it can permanently grant the "Clip" service access rights to music and audio.
[0134] After the electronic device detects that the Always Allow control in FIG. 2E has been activated, as shown in FIG. 2F , user interface 26 displays a prompt message prompting the user to grant the "Clip" service access to photos and videos on the electronic device, a prohibition authorization control, and an Always Allow control. After the electronic device detects that the Always Allow control has been activated, it can permanently grant access rights to the photos and videos to the "Clip" service.
[0135] After the electronic device detects that the always allow control in FIG. 2F is acted upon, the electronic device may start synthesizing the video.
[0136] Of course, if the electronic device is not used for the first time, and if the "one-click filming" function is not enabled for the first time, there is no need to display the user interface shown in Figures 2D-2F. After detecting the user operation acting on the control 235 in Figure 2C, if the user authorization has been obtained, the video synthesis can be started directly.
[0137] The specific process of synthesizing video includes the following:
[0138] First, the electronic device analyzes the original image (also referred to as the source material) corresponding to the thumbnail selected by the electronic device (i.e., the thumbnail included in the edit bar 233), for example, obtaining various information such as size, quantity, and subject matter. During the analysis process, the electronic device may display a user interface 27 shown in FIG2G . User interface 27 includes: a prompt message 271 for notifying the user that the electronic device is analyzing the source material; a progress bar and progress value 272 for notifying the user of the progress of the analysis; and a stop control 273 for interrupting the analysis operation and subsequent video generation.
[0139] After the analysis is complete, the video template is matched to the selected original image. During the matching process, the electronic device may display a user interface 28 shown in Figure 2H. User interface 28 includes: a prompt 281 to inform the user that the electronic device is matching the material; a progress bar and progress value 282 to inform the user of the matching progress; and a stop control 283 to interrupt the matching operation and subsequent video generation.
[0140] The specific process of analysis and matching can be referred to the detailed description of the method embodiment below and will not be described here.
[0141] After the matching is completed, the video generation begins. During the video generation process, the electronic device may display the user interface 29 shown in FIG. 2I .
[0142] As shown in FIG2I , the user interface 29 displays: a return key 291 , a share key 292 , a prompt message 293 , a play / pause control 294 , a video progress bar 295 , and a menu bar 296 at the bottom.
[0143] The return key 291 can be used to return to the user interface 23 shown in FIG. 2C .
[0144] The share button 292 can be used to share the finished video to other applications or contacts.
[0145] Prompt message 293 is used to prompt the user that the video is currently being generated. After the electronic device has made the video, the video can be automatically played in the area 297 where prompt message 293 is located.
[0146] The video progress bar 295 is used to indicate the playing progress and the total length of the video. Before the video is finished, both the playing progress and the total length of the video can be 0.
[0147] The menu bar 296 includes a plurality of controls, such as:
[0148] Template control 296A, which the user can click to view more video templates and select an appropriate template to synthesize the previously selected original image;
[0149] Music control 296B, which the user can click to view available music and synthesize it into the video;
[0150] The export control 296C is used to export the generated video, that is, to store it as a video file. Before the video is generated, the export control 296C may be in a non-clickable state.
[0151] Segment controls 296D, which the user can click to view individual segments in the generated video;
[0152] The editing control 296E is used to edit the generated video, such as cropping, adding filters, etc.
[0153] 2J , which exemplarily shows a frame of user interface 30 displayed when the electronic device automatically plays a video after the video is produced.
[0154] If the user decides to save the video after watching it in area 297, he or she can input a user operation into the export control 296C in the user interface 30. The electronic device can respond to the operation by exporting the video into a certain video format and storing it in the storage area of the electronic device.
[0155] Figure 2K illustrates an example of a user interface 31 displayed during the process of exporting a video by an electronic device. As shown in Figure 2K , the user interface 31 includes: a prompt 311 to notify the user that the electronic device is exporting the video; a progress bar and progress value 312 to indicate the progress of the video export; and a stop control 313 to interrupt the video export operation.
[0156] After the video export is complete, the electronic device may display user interface 32 shown in FIG2L . User interface 32 displays prompt information 321, which informs the user which album in the gallery the generated video has been saved to, and that the video has been saved to the template draft of the "Clip" service. In other words, after the electronic device exports the video, in addition to storing it in the album storage path under the gallery, it may also store it in the template draft storage path under the "Clip" service.
[0157] The prompt information 321 may disappear automatically after being displayed for a period of time, or it may disappear in response to a user operation (eg, the user clicks on "Got it" in FIG. 2L ), after which the electronic device may continue to display a user interface similar to FIG. 2J .
[0158] After the video is stored, the electronic device can view and share the video in the corresponding album in the gallery.
[0159] Figures 2M-2P show the user interface for viewing and sharing videos in the gallery.
[0160] Figure 2M shows a user interface 33 of a gallery displaying albums in an electronic device. The album here refers to the presentation of image classification results in the gallery. As shown in Figure 2M, the user interface 33 displays a thumbnail 331 of the "Video" album and the number of videos contained in the album.
[0161] After the electronic device detects a user operation on a thumbnail in the "Videos" album, it may display user interface 34 shown in FIG2N , which is used to display thumbnails and video durations of each video contained in the "Videos" album. As shown in FIG2N , user interface 34 displays thumbnails 341 of videos previously generated by the electronic device and their corresponding durations.
[0162] The electronic device can detect a user operation on a thumbnail 341 of a video generated by the electronic device in FIG2N , and then display the user interface 35 shown in FIG2O , and play the video in a video playback area 351 of the user interface 35. The user interface 35 also displays various operation controls for the video, such as a progress bar, a share control 352, a favorite control 353, an edit control 354, a delete control 355, and the like.
[0163] As shown in Figure 2O , the electronic device can detect a user action on the share control 352 and, in response to the user action, display the user interface 36 shown in Figure 2P . User interface 36 displays any one or more of the following: one or more application options, one or more contact options, one or more device options, etc. An application option corresponds to an application, for sharing the video to that application; a contact option corresponds to a contact of the electronic device, for sharing the video to the contact's device; and a device option corresponds to a nearby device discovered by the electronic device, for sharing the video to that nearby device.
[0164] For example, as shown in FIG2P , after the electronic device detects a user operation on application option 361, it responds to the operation by sharing the currently displayed video to the application corresponding to application option 361. Of course, the user can also share the video to other applications, other devices, or other contacts according to actual needs.
[0165] FIG3A exemplarily illustrates a process of an image recommendation method provided by the present application, wherein the method is executed by an electronic device.
[0166] As shown in FIG3A , the method may include the following steps:
[0167] S101, an electronic device captures a plurality of images.
[0168] The images captured by the electronic device may be still photos, dynamic photos or videos, which are not limited here.
[0169] For the specific process of the electronic device capturing multiple images, please refer to S301 to S304 in the method shown in FIG3B .
[0170] S102: The electronic device determines the subject, score, and repeatability of each captured image.
[0171] The electronic device can assign a score to each image based on any one or more of its image quality, composition, facial aesthetics, and task action. Repeatability refers to whether an image is identical to previous images taken before it. Theme refers to the scene depicted in the image.
[0172] For a specific implementation of the electronic device determining the theme, score, and repetitiveness of an image, reference may be made to S305 in FIG. 3B .
[0173] S103: The electronic device determines a recommended image from the plurality of images according to the scores, themes, and repetitiveness of the plurality of images.
[0174] For the specific implementation of S103, please refer to S306-S310 in FIG3B .
[0175] S104: The electronic device displays instruction information of the recommended image.
[0176] For the specific implementation of S104, please refer to S311 in FIG3B .
[0177] FIG3B exemplarily shows another process of the image recommendation method provided by the present application, wherein the method is executed by an electronic device.
[0178] As shown in FIG3B , the method may include the following steps:
[0179] S301: The electronic device starts a camera and displays a preview image.
[0180] The electronic device can run a system application or a third-party application to activate the camera to capture images, receive a preview stream reported by the camera, and display the preview image in a preview interface displayed on the display screen based on the preview stream. The electronic device can activate any camera, depending on user needs.
[0181] After the electronic device starts the camera, it can continuously receive the preview stream reported by the camera, so the preview image displayed by the electronic device will also include continuous multiple frames (ie, multiple preview image frames).
[0182] For example, referring to FIG1A-FIG1B, the electronic device may run a camera application and display a preview image frame in the preview window 123 of FIG1B. The user interface 12 shown in FIG1B may be referred to as a preview interface.
[0183] S302: Analyze the current frame preview image from the preview stream and obtain a tag of the preview image. The tag includes the following information: score, theme, and whether it is a transition frame.
[0184] The current preview frame image refers to a preview image frame currently displayed on the display screen. The current preview frame image is also called a preview image frame.
[0185] The electronic device can analyze the current preview frame image from the preview stream.
[0186] When scoring a preview image, one or more of the following factors may be considered:
[0187] 1. Image quality.
[0188] Image quality includes image clarity and / or symmetry. Higher clarity and / or better symmetry indicate higher image quality and a higher score. Symmetry refers to the degree of symmetry within an image (e.g., the left-right symmetry of a building) and can be expressed on a scale of [0, full score].
[0189] 2. The beauty of the face.
[0190] Each image frame includes a subject. The subject of an image refers to an object such as a person or animal that occupies a relatively large portion of the image. An image frame may include one or more subjects. Electronic devices can use a subject detection algorithm, such as a convolutional neural network (CNN), to detect and identify the subject category in an image.
[0191] If the subject of the image is a person or an animal, the image may be scored based on the aesthetics of the face of the person or animal in the image.
[0192] The more beautiful the face in the image is, the higher the score of the image will be.
[0193] 3. The movements of the characters in the image.
[0194] If the image includes a person, the image can be scored based on the excitement of the person's movements. If the person moves and the movement is a preset exciting movement, the image is considered exciting. Preset exciting movements may include jumping, running, raising hands, and raising legs.
[0195] The more exciting the character's movements in the image, the higher the image score.
[0196] 4. Composition.
[0197] Composition refers to the arrangement, layout, and placement of objects within an image. The more coordinated and complete the arrangement and placement of objects, the more beautiful the composition and the higher the image score.
[0198] When scoring the current frame preview image based on any one or more of the above four factors, the weight of each factor on the final score can be the same or different. The weight of each factor on the final score can be preset according to actual needs and is not specifically limited here.
[0199] The following takes the above four factors into consideration as an example to illustrate how to score the preview image of the current frame.
[0200] Formula 1 exemplarily shows the formula for calculating the preview image of the current frame by the electronic device: score=a1*score1+a2*score2+a3*score3+a4*score4 Formula 1
[0201] Where score is the score of the preview image for the current frame, score1 is the score for image quality, score2 is the score for composition, score3 is the score for facial aesthetics, and score4 is the score for character motion. Higher scores indicate better performance in the corresponding dimensions. score1, score2, score3, and score4 are also referred to as the first score, second score, third score, and fourth score, respectively.
[0202] The scoring system for each dimension is consistent. The scoring system can be any scoring system, such as a percentage system or a ten-point system. The score interval of the scoring system, such as [0, 10] on a ten-point scale or [0, 100] on a hundred-point scale, can be referred to as the first score interval. The highest score in the first score interval, such as 10 on a ten-point scale or 100 on a hundred-point scale, can be referred to as the full score.
[0203] In the various implementations of the scoring process described below, high scores include scores whose difference from the full score is less than a preset value, such as 9-10 points on a 10-point scale, such as 90-100 points on a 100-point scale, etc.; low scores include scores whose difference from the full score is greater than a preset value, such as 0-2 points on a 10-point scale, such as 0-20 points on a 100-point scale, etc.
[0204] a1, a2, a3, and a4 are the weights of image quality, composition, facial aesthetics, and character movement on the final score. The sum of a1, a2, a3, and a4 is 1.
[0205] Referring to FIG4 , FIG4 exemplarily shows a process of an electronic device scoring a preview image of a current frame. The process includes the following steps:
[0206] Step 1: Set the initial score1, score2, score3, and score4 to 0.
[0207] Step 2: Calculate the image quality score score1 of the current frame preview image.
[0208] score1 can be calculated based on an image quality assessment (IQA) algorithm.
[0209] Step 3: Determine whether the current frame preview image contains a subject. If not, go directly to step 9; if included, go to step 4.
[0210] Step 4: Calculate the composition score score2 of the current frame preview image.
[0211] Score2 can be calculated based on a composition scoring algorithm. The composition scoring algorithm can be used to score images containing a subject, evaluating the aesthetics of the subject within the overall image. The composition scoring algorithm can be a calculation method based on human skeletal nodes.
[0212] Step 5: Determine the category of the image subject. If it is an animal, go to step 6; if it is a person, go to step 7.
[0213] In step 6, if an animal's face is detected, the facial beauty score score3 is set to a high score. The high score can be preset, for example, a full score (such as 100 points on a 100-point scale) or a value close to the full score (such as 90-100 points on a 100-point scale).
[0214] For animals, as long as the face is captured, the image content is acceptable to the user because a high score can be given for the facial aesthetics. In a specific embodiment, a high score can be given for the facial aesthetics when the front face of the animal is detected.
[0215] A high score can be set as a score whose difference from the full score is less than a preset value.
[0216] Step 7: Calculate the facial beauty score score3 of the character.
[0217] Users often have high expectations for facial aesthetics, so a more detailed calculation method can be used. For example, score3 can be calculated based on the person's expression, facial posture, face size within the image, and the number of faces. Expressions can include whether the person's eyes are open or smiling, while facial posture includes whether the person is facing the camera.
[0218] In one embodiment, score3 may be calculated based on Formula 2:
[0219] In formula 2,
[0220] score3 is the total score of facial beauty, N face is the number of faces, s e,i is the score of the eyes open and closed for the i-th face, s m,i Score of the smile of the i-th face, s d,i is the posture score of the i-th face, s b is the face size influence coefficient, s N is the face ratio coefficient when there are multiple faces, s c is the composition beauty coefficient, s p is the face-free penalty threshold.
[0221] Among them, the score of eyes open and closed is s e,i , Smiley score m,i , Correct posture score s d,i The scoring range of the three factors is within the scoring range of score 3. For example, if score 3 is scored on a percentage basis, the scores of the three factors mentioned above are also within the range of [0,100].
[0222] Face size influence coefficient s b , face ratio coefficient s when there are multiple faces N , composition beauty coefficient s c , no face penalty threshold s p The values are all in the range [0, 1].
[0223] The following describes the various parameters in formula 2:
[0224] Eyes open and closed score e,i: If the character's expression is considered beautiful when the character's eyes are open, the first value is assigned; if the character's expression is considered unbeautiful when the character's eyes are closed, the second value is assigned. The first value is higher than the second value. The first value can be a high score, and the second value can be a low score.
[0225] Smiley score m,i : A smiling person is considered to have a beautiful expression and is assigned the third value; a expressionless person is considered to have an unsightly expression and is assigned the fourth value. The third value is higher than the fourth value. The third value can be a high score, and the fourth value can be a low score.
[0226] Correct posture score d,i : The face facing the camera is considered to be in a correct posture and is assigned the fifth value. If the face is not facing the camera, such as looking up or looking down, the face in the picture is considered to be in an incorrect posture and is assigned the sixth value. The fifth value is higher than the sixth value. The fifth value can be a high score, and the sixth value can be a low score. Face size influence coefficient s b : The face area of appropriate size (such as the face area within the first range) is assigned the seventh value. If the area is too large (such as the face area outside the first range), the face is incomplete and considered unsightly. If the area is too small, the non-subject (possibly a passerby) is photographed as background interference and is considered unsightly. When it is unsightly, the face size influence coefficient s b The eighth value is assigned, and the seventh value is higher than the eighth value. The seventh value can be a higher value in the value range, and the eighth value can be a lower value in the value range.
[0227] Face size influence coefficient s b It can be obtained by querying the three Gaussian curves shown in Figure 5. Specifically, it is calculated based on the part above the intersection of the three Gaussian curves. It represents the square root of the ratio of the face frame area to the image area, and the vertical axis is the face size influence coefficient s b The three curves from left to right represent the face size at far, medium and close distances and the face size influence coefficient s. b This can integrate the subjective evaluation of faces at different distances and limit the scores of faces that are too large or too small.
[0228] Among them, the three Gaussian curves from left to right are:
[0229] μ=0.1278,sigma=9.5Long distance
[0230] μ=0.3819,sigma=3.5 medium distance
[0231] μ=0.65,sigma=9.5Close distance
[0232] The three lines shown in Figure 5 represent the face size at far, medium and close distances and the face size influence coefficient s. bThe Gaussian curve for the mapping relationship is preset. The parameter μ represents the mean, and sig represents the inverse of the standard deviation. The three lines indicate the optimal composition coefficient for the subject at near, medium, and long distances. The peak of the curve represents the optimal position. Using a position above the intersection ensures that the coefficient value maintains a high score within a reasonable distance. w refers to the face frame width, h refers to the face frame height, W refers to the image width, and H refers to the image height. The face frame is the rectangle or square frame created when the electronic device analyzes the current preview frame.
[0233] When there are multiple faces, the proportion coefficient s of each face N :According to the size of each face, the impact of each face on the overall image decreases. N is the preset weight, the sum of which is 1. That is, several face ratio coefficients s are preset in descending order. N , and then correspond to each face according to the area of each face in the image.
[0234] The main character's face must be within a certain range (e.g., the first area) in the center of the image. A face at the edge is considered not the main character, incomplete, and the composition is considered unsightly. Therefore, a penalty factor k is multiplied on the face, reducing its score. The penalty factor k is a preset value. The ninth value is higher than the tenth value. The ninth value can be higher within the range, while the tenth value can be lower within the range.
[0235] Penalty threshold s when there is no face p : A human body is detected but no face is detected. This may be because the face is not in the picture or the person is facing away. This is considered unsightly and the facial beauty score of the corresponding face is reduced by score3.
[0236] According to the calculation method of the above formula 2, score3 is higher when the face posture is correct, the eyes are open and smiling, and the area is moderate. Score3 is higher when all the people in the picture are the main characters.
[0237] The high scores and low scores mentioned above are relative concepts and can be set by electronic devices according to actual needs. This application does not limit their specific values.
[0238] In some implementations, if the subject of the current frame preview image includes both people and animals, the algorithms of step 6 and step 7 may be combined to calculate a comprehensive facial beauty score score3 using weights.
[0239] After step 7, you can go to step 8.
[0240] Step 8: Calculate the character action score score4.
[0241] If the person in the image does not have any exciting movements, score4 is set to a low score (eg, 0).
[0242] If the person in the image performs a compelling action, the confidence score output by the action evaluation algorithm multiplied by a high score (e.g., a perfect score) can be used as the value for score4. The action evaluation algorithm, which can be a CNN algorithm, evaluates the actions of the person in the image and the probability of those actions. The confidence score output by the action evaluation algorithm is the probability that the action in the image is a specific action. The value of score4 is in the range [0, perfect score].
[0243] Step 9: Calculate the score of the preview image of the current frame based on Formula 1.
[0244] Based on the method shown in FIG4 , the current frame preview image is scored. The score of the current frame preview image can reflect the quality of the image, the beauty of the face in the image, the excitement of the character's movements, the composition, etc. The higher the score, the better the image quality, the beauty of the face in the image, the excitement of the character's movements, the composition, etc.
[0245] Of course, the four factors shown in FIG4 are not limited to being considered simultaneously. In some other implementations, only some of the factors may be considered to score the current preview image. If only some of the factors are considered, the weights of the factors may vary, and the sum of the weights is always 1.
[0246] For example, in step 5 above, if the current frame preview image does not include subjects such as animals and people, such as when shooting a landscape, the weight coefficient a3 corresponding to score3 in formula 1 is set to 0, and the remaining weights a1, a2, and a4 are increased accordingly.
[0247] In some embodiments, the electronic device may further analyze whether the current preview image is a transition frame. When a user moves the electronic device quickly, the preview stream reported by the electronic device's camera is a transition frame. Transition frames have low imaging information content and poor image quality, i.e., image frames whose information content and image quality do not meet a first threshold. The electronic device may utilize a transition detection algorithm to identify transition frames.
[0248] In some embodiments, the electronic device may further analyze the subject of the current preview frame. The subject refers to the scene contained in the image, and may include, for example, children, animals, architecture, plants, landscapes, etc. The subject may be obtained based on the content of the image.
[0249] In some embodiments of the present application, the electronic device may use a perception engine to score the current frame preview image and analyze whether it is a transition frame and its theme, etc. The perception engine may be equipped with the various algorithms mentioned above to calculate the required results.
[0250] S303: Perform image encoding on the current frame preview image and save the label of the current frame preview image.
[0251] Image coding, also known as image compression, refers to the technology of representing an image or the information contained in it with fewer bits while meeting certain quality requirements (such as signal-to-noise ratio). This can save storage space on electronic devices.
[0252] In some implementations, the electronic device may encode the tag of the current frame preview image separately into the audio stream of the current frame preview image, and may store it separately from the picture data.
[0253] Before the user enters the command to capture an image, the electronic device's camera will continuously report the preview stream, and during this process the electronic device will continue to display multiple frames of preview images. Therefore, the above steps S302-S303 will be executed multiple times, and the electronic device can save compressed images and labels of the multiple frames of preview images. This saved data can be temporarily stored in the electronic device and stored in a different location from the data saved when the image is actually captured.
[0254] In some embodiments, the electronic device may only store a preset number of compressed preview images and labels, and may discard compressed preview images and labels that are generated earlier than the preset number. This does not affect the subsequent implementation of the present solution and can also save storage space on the electronic device.
[0255] By analyzing the tag information of the preview image during the preview process, the time required for subsequent steps can be saved and the time cost can be reduced. In addition, since the storage capacity of the preview stream is relatively small, the calculation and analysis process can be more convenient.
[0256] S304: Detecting a user operation for capturing an image, and saving a corresponding image in response to the user operation.
[0257] User actions for capturing images can be categorized as follows:
[0258] 1. When the electronic device activates the dynamic photo function, a photo-taking operation is input. In this case, the electronic device stores the preview stream as a dynamic photo. This photo-taking operation can take many forms, such as a touch operation on the display, a voice command, a specific expression, etc. For example, referring to Figure 1B, this photo-taking operation can be a user operation on the photo-taking control 122 when the dynamic photo function of the electronic device is activated.
[0259] For dynamic photos, in addition to analyzing the tags of the preview image before the user's action in S302-S303, it is also necessary to analyze the tags of the preview stream images reported by the camera within a period of time after the user action. For detailed analysis methods, please refer to the previous article. In other words, the electronic device needs to analyze and obtain the tags of each frame in the dynamic photo.
[0260] 2. When the electronic device does not have the dynamic photo function enabled, a photo-taking operation is received. In this case, the electronic device stores the preview stream as a static photo. This photo-taking operation can take many forms, which are not limited here.
[0261] 3. After the electronic device starts the "recording" mode, the received operation of starting recording and the operation of ending recording will store the preview stream during this period as a video.
[0262] It can be seen that the image stored by the electronic device may include one or more preview image frames displayed on the preview interface.
[0263] The image capturing operation in S304 may also be performed multiple times, so that the electronic device can save multiple images according to the user's needs. The multiple images saved by the electronic device may include one or more types, such as dynamic photos, static photos, and videos.
[0264] Exemplarily, referring to FIG. 2A , the user interface 21 displays thumbnails of multiple dynamic photos taken by the electronic device, including thumbnails 211A to 211P.
[0265] Not limited to the electronic device shown in S301-S304 taking images through its own camera, in some other embodiments of the present application, the electronic device can save images sent by other applications other than the camera, or download images from the network, or receive images shared by other electronic devices, and perform subsequent steps based on these images.
[0266] The image stored in the electronic device may be referred to as a first image.
[0267] S305 , obtaining a tag of the stored image, the tag including the following information: subject, score, and whether it is a repeated frame of the stored image.
[0268] The subject of the saved image
[0269] For dynamic photos, the theme that appears most frequently in the multiple frames of the dynamic photo is used as the theme of the dynamic photo. If there are multiple themes with the highest frequency, the theme with the highest priority can be selected from them according to the preset theme priority and used as the theme of the dynamic photo.
[0270] For a still photo, the subject of a frame contained in the still photo is the subject of the still photo.
[0271] For videos, the method for confirming the subject is the same as that for dynamic photos. Please refer to the relevant description.
[0272] Fraction of the saved image
[0273] After saving the image, the electronic device calculates a score for the image.
[0274] The score of the saved image is measured by one or more frames contained in the image. The higher the score of the one or more frames contained in the image, the higher the score of the saved image.
[0275] For dynamic photos, an exemplary scoring method is provided: traverse the image queue in the dynamic photo and find all the frames it contains; optionally, remove the transition frames to obtain the remaining frame sequence FrameSeq [frameStart, ..., frameEnd]. This step can ensure the validity of the final score; then find the frame with the highest score from FrameSeq and record it as the wonderful frame WonderfulFrame. Search for K frames before and after the wonderful frame. If there are less than K frames, fill them with a score of 0; set the weight of the wonderful frame to 1, and the weights of the K frames before and after it decrease by 0.1. Calculate the weighted average score weightAvgScore of the 2K+1 frame scores as the scoring result of the dynamic photo. Remove the transition frames to obtain the image frames contained in the remaining frame sequence FrameSeq, which can also be called the first preview image frame.
[0276] Of course, the above is only an example, and adjustments can be made to the method based on the above example. For example, you can find K frames before the highlight frame, find K' frames different from K after the highlight frame, and calculate the weighted average score of 1+K+K' frames.
[0277] Equivalently, the electronic device can determine the score of the saved image as a weighted average of the wonderful frame, the first preset number of preview image frames before the wonderful frame in the remaining frame sequence FrameSeq, and the second preset number of preview image frames after the wonderful frame in the remaining frame sequence FrameSeq.
[0278] For a still photo, the score of a frame contained in the still photo is the score of the still photo.
[0279] For videos, the score is determined in the same way as for dynamic photos. Please refer to the relevant description.
[0280] Repeatability of saved images
[0281] If there is a historical image that is repeated with the saved image, the saved image is considered to be repeated; if all historical images and the saved image are not repeated, the saved image is considered to be non-repeated.
[0282] The number of historical images can be one or more. The definition of historical images can include any of the following:
[0283] 1. All images stored in the electronic device, such as all images in the gallery.
[0284] 2. Images captured by the electronic device after the camera was last activated.
[0285] 3. Images captured by electronic devices on the same day.
[0286] 4. Images within a fourth preset number recently taken by the electronic device.
[0287] 5. Images captured by an electronic device at a certain location / area after the electronic device enters the location / area.
[0288] The above are only several exemplary definitions of historical photographic images and should not constitute a limitation. The electronic device in this application can define historical photographic images according to actual needs and is not limited here.
[0289] The following example provides a method for determining the repeatability of any of the saved images and historical images:
[0290] Referring to FIG6 , FIG6 exemplarily shows a process of determining the repeatability of two dynamic images. The process may include the following steps:
[0291] Step 1: Set the initial similarity score similarScore to 0.
[0292] Step 2: Determine whether a.imageScore is greater than b.imageScore. If so, go to step 3; if not, go to step 4.
[0293] a represents the image currently saved by the electronic device, and b represents an image in the historical images. a.imageScore is the score of the image saved by the electronic device, and b.imageScore is the score of an image in the historical images.
[0294] Step 3: Set similarScore to 0.
[0295] In this way, the image a with a higher score can be marked as a non-duplicate frame, so that the image a with a higher score is more likely to be recommended in the future, so that the recommendation result is of high quality.
[0296] Step 4: Determine whether one of image a and image b is a motion frame and the other is not a motion frame. If so, go to step 5; if not, go to step 6.
[0297] Whether an image is a motion frame may be determined based on a motion evaluation algorithm.
[0298] Step 5: Set similarScore to 0.
[0299] Step 6: Determine whether both image a and image b are motion frames. If so, go to step 7; if not, go to step 10.
[0300] Step 7: Check whether the time interval between the motion frames in image a and image b is less than the minimum interval intervalMin. If so, go to step 8; if not, go to step 9. The minimum interval intervalMin can be preset as needed.
[0301] Step 8: Set similarScore to 1.
[0302] Step 9: Set similarScore to 0.
[0303] Step 10, determine whether the subjects in image a and image b are the same, if so, go to step 11; if not, go to step 12.
[0304] The same subject here not only refers to the same category, but also means that the subject is the same, such as the same person, the same animal, etc.
[0305] Step 11, set similarScore to 1.
[0306] Step 12: Calculate the similarity between image a and image b using any one of the mean hashing algorithm, Euclidean distance algorithm, cosine distance algorithm, and Hamming distance algorithm, and set similarScore to the calculated value.
[0307] Step 13: determine whether similarScore is greater than a similarity threshold threshold. If so, go to step 14; if not, go to step 15. The similarity threshold threshold can be preset as needed.
[0308] Step 14: Determine whether to save the image as a duplicate image.
[0309] Step 15: Determine that the saved image is a non-duplicate image.
[0310] Since the electronic device can store multiple images, S305 can be executed multiple times to calculate the scores of the multiple stored images respectively.
[0311] In some implementations, the electronic device may encode the tag for saving the image separately into the audio stream of the image, and may store it separately from the picture data.
[0312] S306: A user operation for obtaining a recommended image is detected.
[0313] In some embodiments, the electronic device may generate and display a notification message that indicates that an image is currently available for recommendation, or that a video is currently available for creation. The notification message may be sent from the gallery application to the notification application, which may ultimately display the message. The user operation for obtaining the recommended image may include a user operation (e.g., a click operation, a touch operation, etc.) on the notification message.
[0314] For example, referring to FIG2B , notification message 221 in user interface 22 is one possible implementation. Notification messages are not limited to being displayed in a pull-down notification bar. The electronic device may also display the notification message at the top of the display screen after it is generated, and automatically disappear after a period of time if there is no user interaction.
[0315] The generation and display time of the notification message may include any of the following:
[0316] 1. Generate notification messages periodically, for example, automatically at 4 p.m. or 9 p.m. every day, or at a fixed time every week. That is, a fixed message can be generated and displayed at a preset fixed time.
[0317] 2. Generate a notification message after the electronic device has newly saved a third preset number of images.
[0318] 3. When the electronic device enters a certain location / area, captures multiple images at the location / area, and then leaves the location / area, a notification message is generated. This location / area may be referred to as the first location.
[0319] In other embodiments, a “one-click recommendation” control may be displayed in the user interface of an application that provides an image recommendation function (such as a gallery application or other application), and the user can input user operations on the “one-click recommendation” control to trigger subsequent steps. The user operation acting on the “one-click recommendation” control detected by the electronic device is the user operation for obtaining the recommended image. The embodiments of the present application do not limit in which user interface the “one-click recommendation” control is provided by the application, nor do they limit the form of the “one-click recommendation” control. The name of the “one-click recommendation” control is only an example, and its function has been clearly explained. The control may also be called by other names.
[0320] S307, traverse multiple images and obtain labels of each image.
[0321] The multiple images here can include any of the following:
[0322] 1. All images stored in the electronic device, such as all images in the gallery.
[0323] 2. Images captured between the last time the electronic device turned on the camera and the last time the camera was turned off.
[0324] 3. Images captured by electronic devices on the same day.
[0325] 4. The fifth preset number of images most recently captured by the electronic device.
[0326] 5. Images captured by an electronic device between the time it enters a certain place / area and the time it leaves that place / area.
[0327] The electronic device may extract the labels of the respective images from the memory.
[0328] Exemplarily, referring to FIG. 2A , assuming that thumbnails 211A to 211P are images captured from the last time the electronic device started up its camera to the last time it turned off the camera, the electronic device may extract tags of the original images corresponding to these thumbnails.
[0329] S308 , grouping the multiple images traversed in S307 according to themes.
[0330] For example, if the multiple images include original images corresponding to thumbnails 211A to 211P, the multiple images may be divided into the following groups:
[0331] Landscape group: thumbnail 211A, thumbnail 211B, thumbnail 211F, thumbnail 211J, thumbnail 211L, thumbnail 211N, thumbnail 211O, thumbnail 211P, and corresponding original images.
[0332] Person group: thumbnail 211G, thumbnail 211H, thumbnail 211I, thumbnail 211K, thumbnail 211M, and corresponding original images.
[0333] Cat group: thumbnail 211E, corresponding original image.
[0334] Dog group: thumbnail 211C, thumbnail 211D, and corresponding original images.
[0335] S309: sort the images in each group according to their repetitiveness and scores.
[0336] In some embodiments, for each group of images, non-repeated images are first arranged in front and repeated images are arranged in the back; then, the non-repeated images are sorted from high to low according to the scores, and the repeated images are also sorted from high to low according to the scores.
[0337] In other implementations, each group of images is first sorted from high to low according to the scores. When there are images with the same scores, the non-duplicate images are placed in front and the duplicate images are placed in the back.
[0338] S310 , selecting images with the highest ranking from each group of images as recommended images.
[0339] In some embodiments, the electronic device can select the top K images from different theme groups to form a recommendation sequence. This ensures the diversity and high quality of the recommended images. K can be preset, and its specific value is not limited here.
[0340] In other embodiments, the electronic device may also select different numbers of top-ranked images from different theme groups to form a recommendation sequence. For example, the electronic device may select a certain proportion of the actual number of images in each theme group to form a recommendation sequence.
[0341] In some further embodiments, the electronic device may also select images from only part of the subject group to form a recommendation sequence. For example, the electronic device may select images from a preset subject group.
[0342] The above-mentioned several implementations are only examples. In specific implementations, there may also be other ways of selecting the recommendation sequence.
[0343] After the electronic device obtains multiple recommended images, it can also sort the multiple recommended images. This application does not limit the sorting method. For example, it can be sorted from high to low according to the score, sorted according to a certain theme order, sorted according to the order of storage time, etc.
[0344] S311: Display instruction information of recommended images.
[0345] The indication information of the recommended image may include a thumbnail, or may include a name, a selected check mark box, or other indication information, which is not limited here.
[0346] In some implementations, the electronic device may launch an application that provides an image recommendation function, and then display indication information of the recommended images in a user interface provided by the application.
[0347] Exemplarily, referring to FIG2C , the electronic device may display thumbnails of the multiple images traversed in S307 in the user interface 23 of the gallery application, and mark the images recommended by the electronic device by selecting a selection mark box, and also add numbers in the corresponding selection mark boxes to indicate the ranking of the corresponding images in the recommended images.
[0348] Optionally, S312, adjust the recommended image according to the user operation.
[0349] Adjustment of the recommended images may include: adjusting the order of the multiple recommended images, adding recommended images, deleting recommended images, and the like.
[0350] There are many ways to implement the user operation for adjusting the recommended image, which is not limited here.
[0351] For example, reference may be made to the related textual introduction of FIG. 2C , which exemplarily provides several methods for adjusting the recommended images.
[0352] S313: A user operation for generating a video is detected.
[0353] The user operations used to generate the video can be implemented in various forms, such as user operations on the display screen, voice commands, gestures, expressions, etc., which are not limited here.
[0354] In some implementations, S313 may also be an optional step. The electronic device does not need to receive the user operation and may directly generate a video using the recommended image after determining the recommended image.
[0355] S314: Generate a video based on the recommended image.
[0356] In some embodiments, after the electronic device detects a user operation for generating a video, it may first ask the user for permission to use the "One-click Video" function. For an exemplary user interface for obtaining user authorization, see the user interface shown in Figures 2D-2F, which will not be repeated here.
[0357] The process of generating a video based on the recommended image may include the following steps:
[0358] First, the material (i.e., recommended images) is analyzed, for example, by size, quantity, duration, theme, and other information.
[0359] Then, the audio and video templates are matched according to the material information. The audio and video templates can be divided into video templates and audio templates, or they can be combined into one template. Among them, the video template may include transition effects, filters, image cropping methods, visual elements (such as stickers), etc. The audio template includes audio. The audio and video templates can be stored in the electronic device, or they can be obtained by the electronic device from the cloud server through the network. In some embodiments, after matching the audio and video templates, the user can also adjust it to the form they want according to actual needs.
[0360] The electronic device then generates a video based on the matched audio and video template and recommended material. The order in which the images appear in the generated video can be consistent with the order in which the electronic device determines the images appear in the recommended images. If the recommended images include a dynamic photo, the video can include the full length of the dynamic photo, and the same applies to the video. In other embodiments, the user can select the audio and video template.
[0361] During the above-mentioned process of analyzing materials, matching templates, and generating videos, the electronic device can output some prompt information to remind the user of the steps currently being performed. Examples of these prompt information can be found in the relevant descriptions of Figures 2G-2I.
[0362] In some embodiments, after the electronic device generates a video, it can also automatically play the video so that the user can view the video effect. For example, referring to Figure 2K, user interface 30 shows an effect of automatically playing the video after the video is generated. Of course, the electronic device can also play the video after receiving an operation triggered by the user to play the video.
[0363] S315, the electronic device saves the generated video.
[0364] The electronic device can automatically save the video after generating it.
[0365] Alternatively, the electronic device may also save the video in response to a received user operation after generating the video. For example, the user operation may be a user operation acting on the export control 296C in the user interface 30 shown in FIG. 2J .
[0366] The electronic device can save the video to a storage path under an application that provides image recommendation functions (such as a gallery), and can also store it to a storage path under the "Clip" service, or store it to both paths at the same time, without limitation here.
[0367] S316: A user operation for sharing a video is detected, and the video is shared with a corresponding object.
[0368] After the electronic device saves the generated video, it can display the video in a user interface based on its storage path for the user to view. For example, Figures 2M-2O illustrate a way for the electronic device to display the video.
[0369] After the electronic device detects a user operation for sharing a video, it can share the video with the corresponding object. The user operation for sharing the video can be implemented in various forms, which are not limited here. For example, it can include a user operation acting on the sharing control 352 in Figure 2O. The sharing object can be a contact's device, an application in the electronic device (such as a social platform, memo, notes, etc.), a discovered nearby device, a device where the contact in the application is located, and so on. The application to which the electronic device shares the video can be called the first application, and the device to which the video is shared can be called the first device.
[0370] The electronic device is not limited to displaying the video first and then sharing the video. In some other implementations, the electronic device can also first display various objects, then select one or more objects, and then find the generated video from the gallery or other applications to share with the one or more objects.
[0371] The image recommendation method shown in FIG3B can recommend high-quality, engaging, and dissimilar (i.e., different content) images to users, thereby allowing users to focus on these recommended images. Furthermore, the electronic device can generate videos using the recommended images, meeting the user's need to create high-quality, engaging, and non-repetitive videos. The electronic device can also share the generated videos to social platforms or contacts, meeting the user's need to share videos.
[0372] In the method shown in FIG. 3B , after determining whether an image is a duplicate frame in S305 , the duplicate image can be deleted using the determination result. This deletion can be performed directly by the electronic device, or the electronic device can display the image suggested for deletion to the user, who can then decide whether to delete the duplicate image.
[0373] In the method shown in FIG. 3B , after determining the theme of the image in S305 , the electronic device may further recommend images with the same theme to the user so that the user can view more images with the same theme.
[0374] The electronic device provided by the embodiments of the present application is introduced below.
[0375] 7 shows a schematic diagram of the hardware structure of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 is used to execute the image recommendation method provided in the above method embodiment.
[0376] Electronic device 100 may include a processor 101, memory 102, a wireless communication module 103, a mobile communication module 104, an antenna 103A, an antenna 104A, a power switch 105, a sensor module 106, a focus motor 107, a camera 108, a display screen 109, and the like. Sensor module 106 may include a gyroscope sensor 106A, an acceleration sensor 106B, an ambient light sensor 106C, an image sensor 106D, a distance sensor 106E, and the like. Wireless communication module 103 may include a WLAN communication module, a Bluetooth communication module, and the like. These multiple components may transmit data via a bus.
[0377] The processor 101 may include one or more processing units, for example, the processor 101 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.
[0378] The memory 102 may be used to store computer executable program codes, which may include instructions. The processor 101 executes the instructions stored in the memory 102 to execute various functional applications and data processing of the electronic device 100.
[0379] The wireless communication function of the electronic device 100 can be implemented through the antenna 103A, the antenna 104A, the mobile communication module 104, the wireless communication module 103, the modem processor and the baseband processor.
[0380] Antenna 103A and antenna 104A can be used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.
[0381] The mobile communication module 104 can provide wireless communication solutions, including 2G / 3G / 4G / 5G, for the electronic device 100. The mobile communication module 104 can include at least one filter, a switch, a power amplifier, a low-noise amplifier (LNA), and the like. The mobile communication module 104 can receive electromagnetic waves through the antenna 104A, filter and amplify the received electromagnetic waves, and transmit them to the modem processor for demodulation. The mobile communication module 104 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves for radiation via the antenna 104A.
[0382] A modem processor may include a modulator and a demodulator.
[0383] The wireless communication module 103 can provide wireless communication solutions including wireless local area networks (WLAN), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc. applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 103 receives electromagnetic waves via the antenna 103A, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 101. The wireless communication module 103 can also receive the signal to be sent from the processor 101, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 103A.
[0384] The electronic device 100 can implement a shooting function through an ISP, a camera 108, a video codec, a GPU, a display screen 109, and an application processor.
[0385] The ISP processes data fed back by camera 108. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization for image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 108.
[0386] The camera 108 can be used to capture still or dynamic images or videos. The lens generates an optical image of an object and projects it onto the image sensor. The image sensor converts the optical signal into an electrical signal, which is then transmitted to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or other format. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than one.
[0387] Video codecs are used to compress or decompress digital images. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can open or save pictures or videos in multiple encoding formats.
[0388] Electronic device 100 can implement display functions using a GPU, display screen 109, and an application processor. A GPU is a microprocessor for image processing that connects display screen 109 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 101 may include one or more GPUs that execute program instructions to generate or modify display information.
[0389] The display screen 109 is used to display images, videos, etc. The display screen 109 includes a display panel. In some embodiments, the electronic device 100 may include one or N display screens 109, where N is a positive integer greater than one.
[0390] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0391] In the embodiment of the present application, camera 108 can be used to capture images. The AP or GPU in processor 101 can be used to analyze the preview image of the current frame to obtain a tag for the preview image, analyze user-specified saved images to obtain tags for the images, and determine recommended images from multiple images. The AP or GPU in processor 101 can also be used to generate videos based on the recommended images.
[0392] The memory 102 may be used to store preview images and tags of the preview images, images designated to be saved by the user and tags of the images, and videos generated based on recommended images.
[0393] The display screen 109 may be used to display various types of information to the user, such as the user interface mentioned in the above embodiment.
[0394] Information input devices such as the display screen 109 , the camera 108 , and the microphone can also be used to receive user operations, and the electronic device can execute corresponding functions in response to these user operations.
[0395] The wireless communication module 103 , the mobile communication module 104 , etc. may be used to share the video generated by the electronic device based on the recommended image with other devices.
[0396] The operations performed by the various components in the electronic device 100 may be specifically referred to the relevant description of the above method embodiment, which will not be elaborated here.
[0397] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, a mobile operating system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.
[0398] FIG8 is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application.
[0399] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, a mobile operating system is divided into four layers: application layer, application framework layer / core services layer, system libraries and runtime layer, and kernel layer.
[0400] The application layer can include a series of application packages.
[0401] As shown in Figure 8, the application package may include applications such as camera, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc. The application layer may also include applications for providing image recommendation functions mentioned in the previous embodiment, such as gallery or other third-party applications.
[0402] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.
[0403] As shown in FIG8 , the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.
[0404] A window manager is used to manage window programs.
[0405] Content providers are used to store and retrieve data and make this data accessible to applications.
[0406] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc.
[0407] The phone manager is used to provide the communication function of the electronic device.
[0408] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0409] The notification manager enables applications to display notification information in the status bar, which can be used to convey informational messages and disappear automatically after a short stay without user interaction.
[0410] Runtime can refer to all code libraries, frameworks, etc. required for the program to run.
[0411] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.
[0412] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.
[0413] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0414] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0415] A 2D graphics engine is a drawing engine for 2D drawings.
[0416] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver.
[0417] It should be understood that each step in the above method embodiment can be completed by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.
[0418] The present application also provides an electronic device, which may include a memory and a processor. The memory may be used to store a computer program, and the processor may be used to call the computer program in the memory to enable the electronic device to execute the method executed on the electronic device side in any of the above embodiments.
[0419] The present application also provides an electronic device, which may include a memory and a processor. The memory may be used to store a computer program, and the processor may be used to call the computer program in the memory to enable the electronic device to execute the method executed on the electronic device side in any of the above embodiments.
[0420] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the electronic device side in any of the above embodiments.
[0421] In one possible design, the chip system further includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.
[0422] The chip system can be composed of chips, or can include chips and other discrete devices.
[0423] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.
[0424] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.
[0425] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.
[0426] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method executed by the electronic device side in any of the above embodiments.
[0427] The present application also provides a computer-readable storage medium storing a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device side in any of the above embodiments.
[0428] The various implementation modes of this application can be combined arbitrarily to achieve different technical effects.
[0429] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).
[0430] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.
[0431] In short, the above description is only an embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of this application should be included in the scope of protection of this application.
Claims
1. An image recommendation method, characterized in that: The method is applied to an electronic device, and the method comprises: Displaying a preview interface, wherein the preview interface displays a preview image frame reported by the camera of the electronic device; Detecting a user operation for capturing an image, and saving a first image, wherein the first image includes one or more preview image frames displayed on the preview interface; determining a theme, a score, and a repeatability of the first image, wherein the score of the first image is obtained according to any one or more of image quality, composition, facial beauty, and character action of the first image, and the repeatability indicates whether the first image is repeated with a historically shot image; determining a recommended image from the plurality of first images based on respective scores, themes, and repetitiveness of the plurality of first images; The indication information of the recommended image is displayed.
2. The method according to claim 1, characterized in that The method further comprises: during displaying the preview interface, determining a score of each preview image frame displayed on the preview interface, wherein the score of the preview image frame is obtained according to any one or more of image quality, composition, facial beauty, and character action of the preview image frame; Determining the score of the first image specifically includes: determining the score of the first image according to scores of one or more preview image frames included in the first image.
3. The method according to claim 2, characterized in that Determining the score of each preview image frame displayed on the preview interface specifically includes: determining a first score according to an image quality of a j-th preview image frame, wherein the j-th preview image frame is any preview image frame displayed on the preview interface; Determine whether the j-th preview image frame contains a subject, and if so, determine the second score of the j-th preview image frame according to the composition; if not, set the second score, the third score, and the fourth score of the j-th preview image frame to 0; In the case that the j-th preview image frame contains a subject, determining a third score of the j-th preview image frame according to the subject category of the j-th preview image frame; When the subject included in the j-th preview image frame is a person, determining a fourth score of the j-th preview image frame according to an action of the person; Calculate the weighted average of the first score, the second score, the third score, and the fourth score; The weighted average score is determined as the score of the j-th preview image frame.
4. The method according to claim 3, characterized in that Determining a first score according to the image quality of the j-th preview image frame specifically includes: A first score is determined according to the clarity and / or symmetry of the j-th preview image frame; the higher the clarity and / or symmetry, the higher the first score.
5. The method according to claim 3 or 4, characterized in that: Determining a third score of the jth preview image frame according to the subject category of the jth preview image frame specifically includes: In the case where the subject of the j-th preview image frame includes an animal, if the j-th preview image frame includes the face of the animal, the third score is set to a high score, wherein the high score includes a score whose difference from the full score is less than a preset value, and the full score is the highest score in the scoring range of the third score; or, In the case that the subject of the j-th preview image frame includes a person, the third score is determined based on any one or more of whether the j-th preview image frame includes a face, whether the eyes of the included face are open or closed, the expression of the face, the posture of the face, the area of the face in the j-th preview image frame, the area size, the proportion, and the number of faces.
6. The method according to any one of claims 3 to 5, characterized in that: In the case where the subject of the j-th preview image frame includes a person, determining the third score specifically includes: The third score is determined according to the following formula: is the third score; N face is the number of faces contained in the j-th preview image frame; s e,i is the eye-opening and eye-closing score of the i-th face. When the i-th face opens its eyes, s e,i is set to the first value, when the i-th face closes its eyes, s e,i is set to a second value, the first value being higher than the second value; s m,i is the smile score of the i-th face. When the i-th face smiles, s m,i is set to the third value, when the i-th face is not smiling, s m,i is set to a fourth value, the third value being higher than the fourth value; s d,i is the posture score of the i-th face. When the i-th face faces the camera, s d,i is set to the fifth value, when the i-th face is not facing the camera, s d,i is set to a sixth value, the fifth value being higher than the sixth value; s b is the face size influence coefficient of the i-th face. When the size of the i-th face is within the first range, s b is set to the seventh value, when the size of the i-th face is outside the first range, s b is set to an eighth value, the seventh value being greater than the eighth value; s N is the face proportion coefficient of the i-th face. The larger the area proportion of the i-th face, the larger the N The bigger; s c is the composition beauty coefficient. When the i-th face is located in the first area of the j-th preview image frame, s c is set to the ninth value, when the i-th face is located outside the first area of the j-th preview image frame, s c is set to a tenth value, the ninth value is higher than the tenth value, and the first area includes a central area of the j-th preview image frame; s p is the face-free penalty threshold. When the j-th preview image frame contains a face, s p is 0, and the jth preview image frame does not contain a face, s p Set to the preset faceless penalty threshold.
7. The method according to claim 6, characterized in that The first value, the third value, and the fifth value are all high scores; The second value, the fourth value, and the sixth value are all low scores; The high score includes a score whose difference with the full score is less than a preset value, the low score includes a score whose difference with the full score is greater than a preset value, and the full score is the highest score in the scoring range of the third score.
8. The method according to claim 6 or 7, characterized in that: When the distance between the i-th face and the electronic device is different, the same size of the i-th face corresponds to different s b .
9. The method according to any one of claims 3 to 8, characterized in that: Determining a fourth score of the jth preview image frame according to the action of the character specifically includes: Determining whether the person in the j-th preview image frame includes a preset wonderful action; If so, the fourth score is equal to the product of the confidence level output by the action evaluation output algorithm and the high score; If not, setting the fourth score to a low score; Among them, the high score includes a score whose difference with the full score is less than a preset value, the low score includes a score whose difference with the full score is greater than a preset value, and the full score is the highest score in the scoring range of the fourth score.
10. The method according to any one of claims 2 to 9, characterized in that: Determining a score of the first image according to scores of one or more preview image frames included in the first image specifically includes: The score of the first image is determined according to first preview image frames, wherein the first preview image frames include preview image frames other than transition frames among all preview image frames of the first image, and the transition frames include preview image frames whose information amount and image quality do not reach a first threshold.
11. The method according to claim 10, characterized in that Determining a score of the first image according to the first preview image frame specifically includes: Determine the wonderful frame with the highest score in the first preview image frame; Determine a weighted average score of the wonderful frame, a first preset number of preview image frames before the wonderful frame in the first preview image frame, and a second preset number of preview image frames after the wonderful frame in the first preview image frame; The weighted average score is determined as the score for the first image.
12. The method according to any one of claims 1 to 11, characterized in that: Determining the repeatability of the first image specifically includes: The repeatability of the first image is determined according to any one or more of the scores of the first image and the historically shot image, including whether the person is moving, the subject, and the similarity.
13. The method according to any one of claims 1 to 12, characterized in that Determining the repeatability of the first image specifically includes: If any one of the first image and the historically shot images is repeated, it is determined that the first image is repeated; if the first image and all of the historically shot images are not repeated, it is determined that the first image is not repeated.
14. The method according to claim 12 or 13, characterized in that The method further comprises: Only in any one of the following cases, the first image and the second image in the historically shot images are not repeated, and the second image is any one of the historically shot images: The score of the first image is higher than the score of the second image; Alternatively, the score of the first image is lower than the score of the second image, and one of the first image and the second image is a motion frame, and the other is not a motion frame; Alternatively, the score of the first image is lower than the score of the second image, and both the first image and the second image are motion frames, and the motion time is greater than a preset minimum time interval; Alternatively, the score of the first image is lower than the score of the second image, and both the first image and the second image are non-motion frames, and the subjects are the same, and the similarity between the first image and the second image is lower than a second threshold.
15. The method according to any one of claims 1 to 14, characterized in that: The subject of the first image includes any one of the following: children, animals, buildings, plants, and landscapes; the subject of the first image is determined according to the content of the first image.
16. The method according to any one of claims 1 to 15, characterized in that: Determining a recommended image from the plurality of first images according to respective scores, themes, and repetitiveness of the plurality of first images specifically includes: Dividing the plurality of first images into a plurality of subject groups according to respective subjects of the plurality of first images; For each of the subject groups, the first images are sorted according to the scores and repetitiveness, with the higher scores being arranged first, and the non-repetitive images being arranged first compared to the repetitive images; The first images are screened from each subject group according to the order of precedence to form recommended images.
17. The method according to any one of claims 1 to 16, characterized in that: Before determining a recommended image from the plurality of first images based on the scores, themes, and repetitiveness of the plurality of first images, the method further includes: Displaying a notification message, wherein the notification message is used to prompt the user to view the recommended image; A user operation acting on the notification message is detected.
18. The method according to claim 17, characterized in that The notification message is displayed at any of the following times: Preset fixed time; After a third preset number of images have been captured; After entering a first location, taking a plurality of images at the first location, and after leaving the first location.
19. The method according to any one of claims 1 to 18, characterized in that: The instruction information for displaying the recommended image specifically includes: Thumbnails of the recommended images are displayed.
20. The method according to any one of claims 1 to 19, characterized in that: The instruction information for displaying the recommended image specifically includes: The indication information of the plurality of first images is displayed, and the indication information of the recommended image is selected.
21. The method according to any one of claims 1 to 20, characterized in that: After displaying the indication information of the recommended image, the method further includes: A user action for generating a video is detected; A video is generated according to the recommended image.
22. The method according to claim 21, characterized in that Generating a video according to the recommended image specifically includes: Matching audio and video templates according to the recommended image; A video is generated according to the recommended image and the matching audio and video template.
23. The method according to claim 21 or 22, characterized in that After generating a video according to the recommended image, the method further includes: A user operation for sharing a video is detected; The video is shared to a first application in the electronic device, or the video is shared to a first device different from the electronic device.
24. The method according to any one of claims 1 to 23, characterized in that When the first image is a dynamic photo, saving the first image specifically includes: The moment when the user operation for capturing an image is detected, the moment before and the moment after the moment, the preview image frame displayed on the preview interface, and the sound detected by the electronic device are stored as dynamic photos.
25. An electronic device, characterized in that: include: A memory and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method as described in any one of claims 1-24.
26. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device executes the method as described in any one of claims 1 to 24.
Citation Information
Patent Citations
A data processing method and system in the printing process of a photo printer
CN109634535A
Face recognition method, electronic equipment and computer readable storage medium
CN112036277A
Composition recommendation method and electronic equipment
CN112866557A
Multimedia resource generation method and device, electronic equipment and storage medium
CN113377971A
Photographing method and device for intelligent view-finding recommendation
CN114697530A
Cited By
Building information intelligent recommendation method and system
CN120687687A
