Photo selection methods, devices, equipment and storage media

Abstract

This disclosure provides a method, apparatus, device, and storage medium for photo filtering. It relates to the field of artificial intelligence, particularly to photo classification, photo deduplication, photo filtering, photo set generation, and photo set recommendation. The specific implementation scheme is as follows: N classification labels and their confidence levels are obtained for each photo in a first photo set; a classification feature vector is generated for each photo based on the confidence levels of the N classification labels; the photos in the first photo set are clustered based on their classification feature vectors to obtain multiple cluster categories, and duplicate photos are removed from each cluster category; the deduplicated photos in each cluster category are filtered based on the N classification labels to obtain a second photo set. According to this disclosure, the efficiency of photo deduplication can be improved, thereby increasing the efficiency of generating meaningful photo sets.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, particularly to areas such as photo classification, photo deduplication, photo filtering, photo collection generation, and photo collection recommendation. Background Technology

[0002] With the development and widespread use of smart devices, taking photos to record life has become a habit for many. The idea of ​​automatically organizing and collecting these photos into meaningful story collections (or photo sets) is highly appealing. To achieve automatic organization of these story collections, photo deduplication is the first step. However, current technologies for photo deduplication are relatively inefficient. Summary of the Invention

[0003] This disclosure provides a photo selection method, apparatus, device, and storage medium.

[0004] According to a first aspect of this disclosure, a photo filtering method is provided, comprising:

[0005] Obtain the N category labels and the confidence scores of the N category labels for each photo in the first photo set, where N is a positive integer;

[0006] Based on the confidence scores of the N classification labels corresponding to each photo, a classification feature vector is generated for each photo.

[0007] Based on the classification feature vector of each photo, the photos in the first photo set are clustered to obtain multiple cluster categories, and the photos included in the multiple cluster categories are deduplicated.

[0008] Based on the N category labels corresponding to each photo, the photos in the multiple cluster categories that have undergone deduplication are filtered to obtain the second set of photos.

[0009] According to a second aspect of this disclosure, a photo filtering device is provided, comprising:

[0010] The acquisition module is used to acquire N category labels corresponding to each photo in the first photo set and the confidence scores of the N category labels, where N is a positive integer;

[0011] The generation module is used to generate a classification feature vector for each photo based on the confidence scores of the N classification labels corresponding to each photo.

[0012] The deduplication module is used to cluster the photos in the first photo set based on the classification feature vector of each photo, obtain multiple cluster categories, and deduplicate the photos included in the multiple cluster categories.

[0013] The filtering module is used to filter the deduplicated photos included in the multiple cluster categories based on the N classification labels corresponding to each photo, so as to obtain a second set of photos.

[0014] According to a third aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor; and

[0016] The memory is communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method provided in the first aspect above.

[0018] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method provided in the first aspect above.

[0019] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method provided in the first aspect described above.

[0020] According to the technical solution disclosed herein, the efficiency of deduplication of photos can be improved, thereby improving the efficiency of generating a collection of photos with certain meaning.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a schematic flowchart of a photo selection method according to an embodiment of the present disclosure;

[0024] Figure 2 This is a schematic diagram illustrating the relationship between the subject matter and category tags according to embodiments of this disclosure;

[0025] Figure 3 This is a schematic diagram of the image filtering process according to an embodiment of the present disclosure;

[0026] Figure 4 This is a schematic diagram of the structure of a photo filtering device according to an embodiment of the present disclosure. Figure 1 ;

[0027] Figure 5This is a schematic diagram of the structure of a photo filtering device according to an embodiment of the present disclosure. Figure 2 ;

[0028] Figure 6 This is a schematic diagram of a scene for photo selection according to an embodiment of the present disclosure;

[0029] Figure 7 This is a block diagram of an electronic device used to implement the photo selection method of the embodiments of this disclosure. Detailed Implementation

[0030] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0031] The terms "first," "second," and "third," etc., used in the embodiments, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0032] In related technologies, photo deduplication mainly involves two methods: the first is to use a trained image similarity model for deduplication; the second is to remove unwanted photos based on specified rules such as image size and format. However, the first method has at least the following problems: if there are many photos, calculating the similarity of all photos requires a large amount of computational resources, and the calculated similarity can only be used for photo deduplication, resulting in a very low cost-benefit ratio. The second method has at least the following problems: filtering based on specified rules is difficult to meet the needs of theme-based photo selection. For example, photos of outdoor activities on the same day and photos of indoor video games do not belong to the same theme. If deduplication is performed using specified rules, rules need to be continuously discovered and added, making maintenance difficult.

[0033] This disclosure provides a photo filtering method that can be applied to a photo filtering device. This device has functions such as photo deduplication, photo filtering, or generating a meaningful set of photos. The photo filtering device can be applied to electronic devices, including but not limited to fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers or ordinary servers. Mobile devices include, but are not limited to, vehicle terminals, navigation devices, mobile phones, tablets, etc. In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the photo filtering method includes:

[0034] S101: Obtain the N category labels corresponding to each photo in the first photo set and the confidence scores of the N category labels, where N is a positive integer;

[0035] S102: Generate a classification feature vector for each photo based on the confidence scores of the N classification labels corresponding to each photo;

[0036] S103: Based on the classification feature vector of each photo, perform clustering processing on the photos in the first photo set to obtain multiple cluster categories, and remove duplicate photos included in the multiple cluster categories;

[0037] S104: Based on the N classification labels corresponding to each photo, filter the deduplicated photos included in the multiple cluster categories to obtain the second photo set.

[0038] In this embodiment, the first photo set includes at least two or more photos. In some embodiments, the first photo set may be a collection of photos taken within a certain period of time. For example, the first photo set may be a collection of photos taken within the last week, half a month, or half a year. In other embodiments, the first photo set may be a collection of photos specified by the user. For example, multiple photos uploaded or imported by the user are received and combined into the first photo set. This disclosure does not limit the method of obtaining the first photo set.

[0039] In this embodiment, the N classification labels are obtained using automatic image classification technology. For example, the automatic image classification technology can be based on deep learning. It should be noted that automatic image classification is already a standard function of smart photo albums. To deduplicate photos, the inventors of this application considered reusing the existing image classification function in smart photo albums.

[0040] In this embodiment of the disclosure, N classification labels correspond one-to-one with N confidence scores. For a classification model that supports N classifications, N classification labels TAG1, TAG2, TAG3, ..., TAGN are output for each photo, along with N confidence scores SCORE1, SCORE2, SCORE3, ..., SCOREN for each label; where the confidence score for TAG1 is SCORE1, the confidence score for TAG2 is SCORE2, the confidence score for TAG3 is SCORE3, ..., the confidence score for TAGN is SCOREN.

[0041] In some implementations, a classification feature vector for each photo is generated based on the confidence scores of the N classification labels corresponding to each photo. This includes concatenating the confidence scores of the N classification labels into a single vector and using the concatenated vector as the classification feature vector. For example, the confidence scores of the N classification labels can be concatenated into a single vector in the order of the N classification labels. This provides a computational basis for subsequently calculating the similarity between photos.

[0042] In this embodiment of the disclosure, deduplication includes: removing duplicates from two or more photos whose similarity is greater than a certain threshold. If the similarity between two photos reaches a preset ratio, such as 95%, then one of the two photos is removed.

[0043] In this embodiment of the disclosure, the filtering includes, but is not limited to, topic-based filtering and semantic-based filtering.

[0044] In this embodiment of the disclosure, the number of photos in the second photo set is less than or equal to the number of photos in the first photo set.

[0045] In this embodiment of the disclosure, the second photo set is a photo set with a certain meaning. For example, the second photo set is a photo set with a certain theme. Here, the theme is used to characterize the central idea represented by the photo set. Exemplary themes include, but are not limited to: birthday themes, travel themes, game themes, food themes, cooking themes, dress-up themes, beauty themes, etc. The above is only an illustrative example and is not intended to limit all possible types of themes; it is simply not an exhaustive list.

[0046] The technical solution of this disclosure, compared to the deduplication method using a trained image similarity model, utilizes existing image classification information in the album to obtain N classification labels and the confidence levels of these N classification labels for each photo in the first photo set, and generates a classification feature vector for each photo. Clustering, deduplication, and filtering are then performed based on these feature vectors. This eliminates the need for significant computational resources, improving the efficiency of deduplication and, consequently, photo filtering, and ultimately, the efficiency of generating a meaningful photo set. Furthermore, compared to methods that remove unwanted photos based on specified rules, this solution not only meets the need for theme-based photo filtering but also eliminates the need for continuous rule discovery and addition, making it easier to maintain.

[0047] In some embodiments, deduplication of photos included in multiple cluster categories includes: determining the similarity between photos included in each cluster category; deduplicating two or more photos in each cluster category with a similarity greater than a first threshold according to a deduplication strategy, and retaining one photo from the two or more photos in each cluster category.

[0048] Here, determining the similarity between photos within each cluster category includes: determining the similarity between photos based on the classification feature vectors of each photo within each cluster category. For example, the similarity between photos can be determined based on the Euclidean distance between the classification feature vectors. Alternatively, the similarity can be determined based on the cosine distance between the classification feature vectors.

[0049] In one example, the deduplication strategy includes randomly selecting photos to be removed. For instance, cluster 1 includes 10 photos, denoted as photo 1, photo 2, ..., photo 10. If the similarity between photo 1 and photo 10 is greater than a first threshold, then one of photo 1 and photo 10 is randomly removed.

[0050] In another example, the deduplication strategy includes deduplication according to timeline. Generally, photos taken earlier are removed. For example, cluster category 1 includes 10 photos, labeled as photo 1, photo 2, ..., photo 10. Photo 1 and photo 10 have a similarity greater than a first threshold. If photo 1 was taken earlier than photo 10, then photo 1 is removed.

[0051] In another example, the deduplication strategy includes deduplication by sharpness. Generally, photos with poor sharpness are removed. For example, cluster category 1 includes 10 photos, denoted as photo 1, photo 2, ..., photo 10. Photo 1 and photo 10 have a similarity greater than a first threshold. If the sharpness of photo 1 is lower than that of photo 10, then photo 1 is removed.

[0052] Here, the first threshold is a pre-determined threshold based on a known set of historical photographs. This set of historical photographs is a known collection of photographs with a clear theme, capable of conveying a specific story or event. The first threshold can also be set or adjusted according to design requirements such as deduplication accuracy or deduplication speed.

[0053] Thus, compared to deduplication using a trained image similarity model, determining the similarity between photos based on the classification feature vectors of each photo in each cluster category, and then deduplicating based on the similarity between photos, saves computational resources and can quickly determine the classification feature vectors of each photo using the existing image classification information in the album. This helps to quickly determine the similarity between photos and improves the deduplication efficiency.

[0054] In some embodiments, based on the N classification labels corresponding to each photo, the deduplicated photos included in multiple cluster categories are filtered, including: based on the N classification labels corresponding to each photo in multiple cluster categories, the top K topics of the multiple cluster categories are counted, where K is a positive integer; the co-occurrence probability between the top K topics of any two cluster categories is determined; if the co-occurrence probability between the top K topics of any two cluster categories is less than a second threshold, one of the cluster categories is retained according to the filtering strategy.

[0055] Here, the value of K can be set or adjusted according to design requirements such as screening accuracy or screening speed. For example, K = 1 to 3.

[0056] Here, co-occurrence probability refers to the probability of both events occurring together.

[0057] In some implementations, for each photo included in each cluster category with N category labels, the top K topics included in each cluster category are counted, including: for each photo included in each cluster category with N category labels, the top K category labels with the highest confidence level included in each cluster category are counted, and the topics represented by the K category labels of each cluster category are taken as the top K topics included in each cluster category; where K is less than N.

[0058] For example, cluster category 1 includes M photos, denoted as M1, M2, ..., Mm. Each photo has N classification labels. The classification labels with a confidence level greater than a certain threshold, such as 0.9, are counted for each photo. From the classification labels with a confidence level greater than a certain threshold, such as 0.9, the top K labels are determined. The themes represented by the top K labels are taken as the top K themes included in cluster category 1.

[0059] In other implementations, for each photo in each cluster category with N category labels, the top K topics in each cluster category are counted, including: counting the R category labels with the highest confidence level for each photo in each cluster category, resulting in R×N category labels for each cluster category; and taking the topics represented by the top K category labels with the highest percentage among the R×N category labels as the top K topics for each cluster category; wherein R is greater than or equal to 1 and less than N, and K is less than N.

[0060] For example, cluster category 1 includes M photos, denoted as M1, M2, ..., Mm. Each photo has N category labels. The top R category labels with the highest confidence in each photo in cluster category 1 are counted. The R×N category labels included in cluster category 1 are counted. The topics represented by the top K category labels with the highest proportion among the R×N category labels are taken as the top K topics of cluster category 1, where R is less than N and R×N is greater than K.

[0061] Here, the second threshold is a threshold predetermined based on a known set of historical photos. The second threshold can also be set or adjusted according to design requirements such as filtering precision or filtering speed.

[0062] In one example, the filtering strategy includes random filtering. For instance, the first set of photos is divided into 10 clusters, denoted as cluster 1, cluster 2, ..., cluster 10. If the theme of clusters 1, 2, 5, 7, and 9 is theme 1, the theme of clusters 3, 4, and 10 is theme 2, and the theme of clusters 6 and 8 is theme 3, then photos belonging to the same theme, such as theme 2, are randomly selected. That is, photos belonging to clusters 3, 4, and 10 are used as the second set of photos.

[0063] In another example, the filtering strategy includes filtering by topic. Generally, photos on a target topic are filtered out. For example, the first photo set is divided into 10 clusters, denoted as cluster 1, cluster 2, ..., cluster 10. The topics of clusters 1, 2, 5, 7, and 9 are topic 1; the topics of clusters 3, 4, and 10 are topic 2; and the topics of clusters 6 and 8 are topic 3. If the target topic is topic 1, then photos included in clusters 1, 2, 5, 7, and 9 are filtered out to form the second photo set.

[0064] This improves the efficiency of photo selection, and consequently, the efficiency of generating meaningful photo collections.

[0065] In some embodiments, retaining one of the two cluster categories according to a screening strategy includes: retaining the cluster category with a higher co-occurrence probability with other cluster categories.

[0066] For example, the first set of photos is clustered into three clusters, denoted as cluster category 1, cluster category 2, and cluster category 3. If the co-occurrence probability between cluster category 1 and cluster category 2 is less than a second threshold, the co-occurrence probability between cluster category 1 and cluster category 3 is 0.8, and the co-occurrence probability between cluster category 2 and cluster category 3 is 0.6. In this case, cluster category 1 is retained between cluster category 1 and cluster category 2. The resulting second set of photos includes cluster category 1 and cluster category 3.

[0067] This makes the photos in the generated second set of photos more relevant and improves the accuracy of the generated set of photos with certain significance.

[0068] In some embodiments, the above photo filtering method may further include: calculating the co-occurrence probability of a first theme and a second theme in multiple historical photo sets; and determining a second threshold for the first theme and the second theme based on the co-occurrence probability of the first theme and the second theme.

[0069] Here, a historical photo collection is a known collection of photos with a clear theme that can tell a certain story or event.

[0070] Thus, by statistically analyzing the co-occurrence probabilities of the first and second themes in multiple historical photo sets, we can identify which themes are strongly correlated and which are weakly correlated, which helps determine a more reasonable second threshold and improves the accuracy of photo selection.

[0071] In some embodiments, calculating the co-occurrence probability of a first theme and a second theme in multiple historical photo sets includes: calculating the number of sets in multiple historical photo sets that simultaneously contain a first category label and a second category label, wherein the first category label is a label included in the first theme and the second category label is a label included in the second theme; calculating the total number of sets in multiple historical photo sets that contain either the first category label or the second category label; and obtaining the co-occurrence probability of the first theme and the second theme based on the number of sets and the total number of sets.

[0072] Here, the co-occurrence probability of the first topic and the second topic is calculated as follows: Co-occurrence probability of the first topic and the second topic = (Number of historical photo sets that simultaneously have the first category label and the second category label) / Total number of historical photo sets that have either the first category label or the second category label.

[0073] In this way, the probability of different themes appearing together can be determined by using the classification tags of photos. Since there is no need for a special theme recognition model to identify the theme of the photo set, the efficiency of photo screening is improved.

[0074] In some embodiments, based on the N classification labels corresponding to each photo, the photos included in multiple cluster categories after deduplication are filtered to obtain a second photo set, including: determining cluster categories that are not related to the target theme based on the N classification labels corresponding to each photo; removing photos included in the cluster categories that are not related to the target theme from the first photo set to obtain the second photo set.

[0075] In some embodiments, a topic may correspond to multiple category tags. Figure 2 This diagram illustrates the relationship between topics and category tags, such as... Figure 2 As shown, different topics can correspond to different category tags. For example, topic 1 corresponds to two category tags: category tag 1 and category tag 3. Topic 2 corresponds to three category tags: category tag 2, category tag N-1, and category tag N. Topic X corresponds to only one category tag: category tag N. It should be understood that... Figure 3 The diagram illustrating the relationship between topics and category tags is for illustrative purposes only and not restrictive.

[0076] For example, the first set of photos is divided into 10 clusters, denoted as cluster 1, cluster 2, ..., cluster 10. The theme of cluster 1, cluster 2, cluster 5, cluster 7 and cluster 9 is theme 1; the theme of cluster 3, cluster 4 and cluster 10 is theme 2; and the theme of cluster 6 and cluster 8 is theme 3. If the target theme is theme 1, then cluster 3, cluster 4, cluster 10, cluster 6 and cluster 8 are removed, and the photos included in cluster 1, cluster 2, cluster 5, cluster 7 and cluster 9 are used as the second set of photos.

[0077] In this way, a collection of photos that matches the target theme can be selected, satisfying the customization needs of photo collection generation.

[0078] In some embodiments, based on the N classification labels corresponding to each photo, the photos included in multiple cluster categories after deduplication are filtered to obtain a second photo set, including: determining the number of groups of semantically related cluster categories in the first photo set; dividing the photos included in the cluster categories belonging to the same group into a set to obtain a second photo set with a number equal to the number of groups.

[0079] For example, the first set of photos is divided into 10 clusters, denoted as cluster 1, cluster 2, ..., cluster 10. If clusters 1, 2, 5, 7, and 9 are semantically related and belong to theme 1; clusters 3, 4, and 10 are semantically related and belong to theme 2; and clusters 6 and 8 are semantically related and belong to theme 3, then the first set of photos is divided into three groups: the group belonging to theme 1, the group belonging to theme 2, and the group belonging to theme 3. This further divides the first set of photos into three second sets. Specifically, the first second set includes photos from clusters 1, 2, 5, 7, and 9; the second second set includes photos from clusters 3, 4, and 10; and the third second set includes photos from clusters 6 and 8.

[0080] In this way, the first set of photos can be divided into multiple sets of photos with certain meanings, which improves the efficiency of photo selection and also increases the diversity of the generated second set of photos.

[0081] To deduplicate images, existing image category information in the smart album can also be reused. Figure 3 The flowchart for image filtering is shown, such as... Figure 3 As shown, the process includes:

[0082] S301: Obtain the initial image set;

[0083] S302: Generate a set of classification feature vectors;

[0084] S303: Perform feature clustering;

[0085] S304: Perform similarity deduplication on images within the same cluster category;

[0086] S305: Perform semantic filtering on images of different cluster categories;

[0087] S306: Output the filtered image set.

[0088] Here, each image in the initial image set has N category labels, specifically denoted as TAG1, TAG2, TAG3...TAGN. Furthermore, each category label has a confidence score, and each image has N confidence scores, specifically denoted as SCORE1, SCORE2, SCORE3...SCOREN. These N confidence scores can be concatenated into a single classification feature vector. Based on the classification feature vectors of each image in the initial image set, a set of classification feature vectors is generated. The similarity between images is determined based on the Euclidean distance or cosine distance between the classification feature vectors in the set.

[0089] Image category tags can represent at least part of the image's theme information. Based on statistical methods, the co-occurrence probability P(T1, T2) of any two tags TAG1 and TAG2 in an existing image set (to ensure effectiveness, sets with good user feedback can be used) is calculated as: (Number of image sets where TAG1 and TAG2 appear simultaneously) / Total number of image sets. This co-occurrence probability can then be used when filtering new stories by theme. When it is below a certain threshold, it indicates that the two images do not co-occur on the same theme and need to be removed, thus achieving the effect of theme filtering.

[0090] In practical implementation, to improve efficiency, the initial image set can be clustered using the image classification feature vectors. For images within the same cluster, pairwise comparisons are made. If the distance is less than a certain threshold, the two images are considered highly similar and likely belong to a series of photos, thus they can be removed. For images in different clusters, their themes differ. The top-K themes of each cluster are counted (K is typically 1-3), and the co-occurrence probability between the top themes of two clusters is evaluated. If the co-occurrence probabilities are both low, the two clusters are considered semantically unco-occurring, and one cluster can be removed. For example, the clusters of "snow mountain" and "beach" do not have semantic co-occurrence.

[0091] In this way, the image classification features already present in most smart photo albums can be used for similarity deduplication, saving the effort of calculating an additional similarity feature. This can significantly reduce computational costs for products with a large number of images. Simultaneously, by utilizing the semantic features inherent in image classification, the co-occurrence probability between categories can be obtained based on existing image set information, allowing for further semantic filtering to remove images incompatible with the theme of the image set.

[0092] It should be understood that Figure 3 The flowchart shown is merely illustrative and not limiting, and it is extensible; those skilled in the art can use it as a basis. Figure 3Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0093] This disclosure provides a photo filtering device, such as... Figure 4 As shown, the photo filtering device may include: an acquisition module 401, used to acquire N classification labels and the confidence scores of the N classification labels for each photo in a first photo set, where N is a positive integer; a generation module 402, used to generate a classification feature vector for each photo based on the confidence scores of the N classification labels for each photo; a deduplication module 403, used to perform clustering processing on the photos in the first photo set based on the classification feature vectors of each photo to obtain multiple cluster categories, and to deduplicate the photos included in the multiple cluster categories; and a filtering module 404, used to filter the photos included in the multiple cluster categories after deduplication based on the N classification labels for each photo to obtain a second photo set.

[0094] In some embodiments, the deduplication module 403 includes: a first determining submodule, configured to determine the similarity between photos included in each of the multiple cluster categories; and a deduplication submodule, configured to deduplicatize two or more photos in each cluster category whose similarity is greater than a first threshold according to a deduplication strategy, and retain one photo from the two or more photos in each cluster category.

[0095] In some embodiments, the filtering module 404 includes: a first statistical submodule, configured to count the top K topics of multiple cluster categories based on N classification labels corresponding to each photo in multiple cluster categories, where K is a positive integer; a second determination submodule, configured to determine the co-occurrence probability between the top K topics of any two cluster categories; and a first filtering submodule, configured to retain one of the two cluster categories according to a filtering strategy if the co-occurrence probability between the top K topics of any two cluster categories is less than a second threshold.

[0096] In some embodiments, the first filtering submodule is configured to: retain the cluster category with a higher co-occurrence probability with other cluster categories among any two cluster categories.

[0097] In some embodiments, such as Figure 5 As shown, the photo filtering device may further include: a preprocessing module 405, used to calculate the co-occurrence probability of a first theme and a second theme in multiple historical photo sets; and a determination module 406, used to determine a second threshold for the first theme and the second theme based on the co-occurrence probability of the first theme and the second theme.

[0098] In some embodiments, the preprocessing module 405 includes: a second statistics submodule, used to count the number of sets in multiple historical photo sets that simultaneously contain a first category label and a second category label, wherein the first category label is a label included in a first theme, and the second category label is a label included in a second theme; a third statistics submodule, used to count the total number of sets in multiple historical photo sets that contain either the first category label or the second category label; and a third determination submodule, used to obtain the co-occurrence probability of the first theme and the second theme based on the number of sets and the total number of sets.

[0099] In some embodiments, the filtering module 404 includes: a second filtering submodule, configured to determine cluster categories unrelated to the target theme based on N classification labels corresponding to each photo; and to remove photos included in the cluster categories unrelated to the target theme from the first photo set to obtain a second photo set.

[0100] In some embodiments, the filtering module 404 includes: a third filtering submodule, configured to determine the number of groups of semantically related cluster categories in the first photo set; and to divide the photos included in the cluster categories belonging to the same group into a set to obtain a second photo set with a number equal to the number of groups.

[0101] Those skilled in the art should understand that the functions of each processing module in the photo filtering device of this disclosure embodiment can be understood with reference to the relevant description of the aforementioned photo filtering method. Each processing module in the photo filtering device of this disclosure embodiment can be implemented by an analog circuit that implements the functions described in the embodiments of this disclosure, or by running software that performs the functions described in the embodiments of this disclosure on an electronic device.

[0102] The photo filtering apparatus of this disclosure improves the efficiency of deduplication, thereby improving the efficiency of photo filtering and the efficiency of generating a meaningful photo set. Furthermore, compared to methods that remove unwanted photos based on specified rules, it not only meets the need for theme-based photo filtering but also eliminates the need for continuously discovering and adding rules, making it easier to maintain.

[0103] Figure 6 This shows a schematic diagram of a photo filtering scenario, from Figure 6 As can be seen, electronic devices such as cloud servers receive a first set of photos from various terminals; for each first set of photos, photos are filtered to generate one or more second sets of photos corresponding to the first set. In this way, image similarity deduplication and semantic deduplication are achieved through classification features, thus completing the photo filtering task in the photo set generation.

[0104] Here are a few examples of photo filtering scenarios. For instance, sending photos from a storage device to an electronic device allows the device to automatically filter the photos and generate multiple meaningful photo sets. Another example is inputting a target theme, such as a birthday theme, so the electronic device can automatically filter the photos and generate a set of photos related to that theme. Yet another example is issuing a deduplication instruction so the electronic device can automatically deduplicate the stored photos.

[0105] It should be understood that Figure 6 The scene diagrams shown are merely illustrative and not restrictive; those skilled in the art can interpret them based on... Figure 6 Even with various obvious changes and / or substitutions to the examples, the resulting technical solutions still fall within the scope of this disclosure.

[0106] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0108] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0109] like Figure 7 As shown, device 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 702 or a computer program loaded from storage unit 708 into random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0110] Multiple components in device 700 are connected to I / O interface 705, including: input unit 706, such as keyboard, mouse, etc.; output unit 707, such as various types of monitors, speakers, etc.; storage unit 708, such as disk, optical disk, etc.; and communication unit 709, such as network card, modem, wireless transceiver, etc. Communication unit 709 allows device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0111] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the photo selection method. For example, in some embodiments, the photo selection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of the photo selection method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the photo filtering method by any other suitable means (e.g., by means of firmware).

[0112] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM), flash memory, optical fiber, compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0116] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0117] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0118] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0119] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A photo filtering method, comprising: Obtain the N category labels corresponding to each photo in the first photo set and the confidence scores of the N category labels, where N is a positive integer; Based on the confidence scores of the N classification labels corresponding to each photo, a classification feature vector is generated for each photo. Based on the classification feature vector of each photo, the photos in the first photo set are clustered to obtain multiple cluster categories, and the photos included in the multiple cluster categories are deduplicated. Based on the N classification labels corresponding to each photo, the deduplicated photos included in the multiple cluster categories are filtered to obtain a second photo set; The step of filtering the deduplicated photos included in the multiple clustering categories based on the N classification labels corresponding to each photo includes: Based on the N category labels corresponding to each photo in the multiple cluster categories, the top K topics of the multiple cluster categories are counted, where K is a positive integer; Determine the co-occurrence probability between the top K topics in any two clusters among the plurality of cluster categories; If the co-occurrence probability between the top K topics in any two cluster categories is less than a second threshold, one of the two cluster categories is retained according to the screening strategy.

2. The method according to claim 1, wherein, The process of deduplicating photos included in the multiple clustering categories includes: Determine the similarity between the photos included in each of the multiple cluster categories; For two or more photos in each cluster category with a similarity greater than the first threshold, deduplication is performed according to the deduplication strategy, and one photo is retained from the two or more photos in each cluster category.

3. The method according to claim 1, wherein, The step of retaining one of the two cluster categories according to the screening strategy includes: Retain the cluster category with the higher co-occurrence probability with other cluster categories from any two cluster categories.

4. The method according to claim 1, further comprising: The co-occurrence probability of the first and second themes in multiple sets of historical photos was calculated. Based on the co-occurrence probability of the first topic and the second topic, a second threshold is determined for the first topic and the second topic.

5. The method according to claim 4, wherein calculating the co-occurrence probability of the first theme and the second theme in multiple sets of historical photos includes: The number of sets in multiple historical photo collections that simultaneously contain both a first category label and a second category label is counted. Here, the first category label is a label included in a first theme, and the second category label is a label included in a second theme. The total number of sets containing either the first or second category label was counted among multiple historical photo collections; Based on the number of sets and the total number of sets, the co-occurrence probability of the first topic and the second topic is obtained.

6. The method according to claim 1, wherein, The second photo set is obtained by filtering the deduplicated photos included in the multiple clustering categories based on the N classification labels corresponding to each photo, including: Based on the N classification labels corresponding to each photo, determine the cluster categories that are not related to the target topic; Photos in the first photo set that are not related to the target topic are removed to obtain the second photo set.

7. The method according to claim 1, wherein, The second photo set is obtained by filtering the deduplicated photos included in the multiple clustering categories based on the N classification labels corresponding to each photo, including: Determine the number of semantically related cluster categories in the first photo set; Photos belonging to the same cluster category are grouped into a set, resulting in a second set of photos equal to the number of the clusters.

8. A photo filtering device, comprising: The acquisition module is used to acquire N classification labels corresponding to each photo in the first photo set and the confidence level of the N classification labels, where N is a positive integer; The generation module is used to generate a classification feature vector for each photo based on the confidence level of the N classification labels corresponding to each photo. The deduplication module is used to perform clustering processing on the photos in the first photo set based on the classification feature vector of each photo to obtain multiple cluster categories, and to deduplicate the photos included in the multiple cluster categories; The filtering module is used to filter the deduplicated photos included in the multiple cluster categories based on the N classification labels corresponding to each photo, so as to obtain a second set of photos. The filtering module includes: The first statistical submodule is used to count the top K topics of each of the multiple cluster categories based on the N classification labels corresponding to each photo in the multiple cluster categories, where K is a positive integer; The second determining submodule is used to determine the co-occurrence probability between the top K topics of any two cluster categories among the plurality of cluster categories; The first filtering submodule is used to retain one of the two cluster categories according to a filtering strategy if the co-occurrence probability between the top K topics of any two cluster categories is less than a second threshold.

9. The apparatus according to claim 8, wherein, The deduplication module includes: The first determining submodule is used to determine the similarity between the photos included in each of the multiple clustering categories; The deduplication submodule is used to deduplicatize two or more photos with a similarity greater than a first threshold in each cluster category according to the deduplication strategy, and retain one photo from the two or more photos in each cluster category.

10. The apparatus according to claim 8, wherein, The filtering submodule is used for: Retain the cluster category with the higher co-occurrence probability with other cluster categories from any two cluster categories.

11. The apparatus according to claim 8, further comprising: The preprocessing module is used to calculate the co-occurrence probability of the first and second themes in multiple historical photo sets; The determination module is used to determine a second threshold for the first topic and the second topic based on the co-occurrence probability of the first topic and the second topic.

12. The apparatus according to claim 11, wherein the preprocessing module comprises: The second statistics submodule is used to count the number of sets in multiple historical photo sets that simultaneously contain both the first category label and the second category label, wherein the first category label is a label included in the first topic, and the second category label is a label included in the second topic. The third statistics submodule is used to count the total number of sets in which the first category label or the second category label appears in multiple historical photo sets; The third determining submodule is used to obtain the co-occurrence probability of the first topic and the second topic based on the number of sets and the total number of sets.

13. The apparatus according to claim 8, wherein, The filtering module includes: The second filtering submodule is used for: Based on the N classification labels corresponding to each photo, determine the cluster categories that are not related to the target topic; Photos in the first photo set that are not related to the target topic are removed to obtain the second photo set.

14. The apparatus according to claim 8, wherein, The filtering module includes: The third filtering submodule is used for: Determine the number of semantically related cluster categories in the first photo set; Photos belonging to the same cluster category are grouped into a set, resulting in a second set of photos equal to the number of the clusters.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images