Method and device for querying picture, electronic equipment and storage medium

Abstract

The present disclosure provides a method and device for querying pictures, electronic equipment and storage medium, relates to the field of artificial intelligence, in particular to the field of image recognition and big data technology, and can be applied in the field of intelligent cloud, smart city and urban security. The specific implementation scheme is as follows: in response to receiving a to-be-queried picture, recalling M candidate picture sets from a plurality of picture sets according to the to-be-queried picture, wherein each of the plurality of picture sets comprises at least one sub-set, and each of the at least one sub-set comprises at least one picture; determining N sub-sets in the M candidate picture sets according to the corresponding relationship between the picture sets and the sub-sets; and determining a target sub-set from the N sub-sets as a target picture set according to at least one of a first similarity and a second similarity corresponding to each of the N sub-sets, wherein 1<=M<=N, the first similarity is the similarity between the sub-set and the to-be-queried picture, and the second similarity is the similarity between the sub-set and a reference sub-set.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, particularly to the fields of image recognition and big data technology, and can be applied in scenarios such as smart cloud, smart city, and urban security. Background Technology

[0002] The construction of smart cities represents a new concept in modern urban governance. In some cities, surveillance cameras can be used to create an image set for each object within the city. For example, in security scenarios, an image of a target object can be used to search for the corresponding image set in a database storing multiple image sets. The images in this set can reflect the object's behavioral trajectory, thus facilitating the understanding of the object's information. Summary of the Invention

[0003] This disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for querying images.

[0004] According to one aspect of this disclosure, a method for querying an image is provided, comprising: in response to receiving an image to be queried, retrieving M candidate image sets from multiple image sets based on the image to be queried, each of the multiple image sets including at least one subset, and each of the at least one subset including at least one image; M being an integer greater than or equal to 1; determining N subsets from the M candidate image sets based on the correspondence between image sets and subsets, where N is an integer greater than or equal to M; and determining a target subset from the N subsets as the target image set corresponding to the image to be queried based on at least one of a first similarity and a second similarity corresponding to each of the N subsets; wherein the first similarity is the similarity between each of the N subsets and the image to be queried, and the second similarity is the similarity between each of the N subsets and a reference subset, and the reference subset is the subset among the N subsets corresponding to the largest first similarity.

[0005] According to another aspect of this disclosure, an apparatus for querying images is provided, including a recall module, a first determination module, and a second determination module. The recall module is configured to, in response to receiving an image to be queried, recall M candidate image sets from multiple image sets, each of the multiple image sets including at least one subset, and each at least one subset including at least one image; M is an integer greater than or equal to 1. The first determination module is configured to determine N subsets from the M candidate image sets based on the correspondence between image sets and subsets, where N is an integer greater than or equal to M. The second determination module is configured to determine a target subset from the N subsets as the target image set corresponding to the image to be queried, based on at least one of a first similarity and a second similarity corresponding to each of the N subsets; wherein the first similarity is the similarity between each of the N subsets and the image to be queried, and the second similarity is the similarity between each of the N subsets and a reference subset, and the reference subset is the subset among the N subsets corresponding to the largest first similarity.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods provided in this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods provided in this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the methods provided in this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of the method and apparatus for querying images according to embodiments of this disclosure;

[0012] Figure 2 This is a schematic flowchart of a method for querying images according to an embodiment of the present disclosure;

[0013] Figure 3This is a schematic diagram illustrating the construction of an image set according to an embodiment of the present disclosure;

[0014] Figure 4 This is a schematic diagram illustrating the construction of a database using an image set according to an embodiment of this disclosure;

[0015] Figure 5 This is a schematic flowchart of a recall operation according to an embodiment of the present disclosure;

[0016] Figure 6 This is a schematic flowchart illustrating the determination of a target image set according to an embodiment of the present disclosure;

[0017] Figure 7 This is a schematic diagram illustrating the principle of a method for querying images according to an embodiment of this disclosure;

[0018] Figure 8 This is a schematic structural block diagram of a device for querying images according to embodiments of the present disclosure; and

[0019] Figure 9 This is a structural block diagram of an electronic device used to implement the image querying method of the embodiments of this disclosure. Detailed Implementation

[0020] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0021] In some technical solutions, cameras in the city can be used to capture images of objects, and then the multiple images can be archived as objects. For example, the features of each image captured by the camera can be represented, and then based on these features, clustering methods can be used to cluster the images into multiple image sets. Each image set can correspond to one object, and the image sets can be stored in a database. Clustering methods used in the clustering process can include, for example, k-means clustering and DBSCAN (Density-Based Spatial Clustering of Applications with Noise).

[0022] In security scenarios, for example, a search can be conducted using the image to be queried to find a target image set in a database. For instance, if an image set contains 100 images, each with 100 corresponding features, the average of these 100 features can be calculated and used as the centroid of the image set. Then, the similarity between the centroid of each image set and the features of the image to be queried can be calculated. The image set with the highest similarity is identified as the target image set and returned to the client as the query result. Images in the target image set can then be used to understand information about the target object.

[0023] Understandably, during the image collection construction process, due to factors such as shooting angle and lighting, it's impossible to archive all images of the same object into a single collection. This leads to mixed and scattered images within the collection. Mixed collection means that images of multiple objects are categorized into the same collection, which affects accuracy. Scattered collection means that multiple images of the same object are scattered across different collections, which affects recall. It can be seen that both mixed and scattered collections contribute to errors in the target image collection.

[0024] The present disclosure aims to provide a method and apparatus for querying images, which can improve the recall and accuracy of a target image set, thereby presenting more comprehensive and accurate images.

[0025] The technical solutions provided in this disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Figure 1 This is a schematic diagram illustrating an application scenario of the method and apparatus for querying images according to embodiments of this disclosure.

[0027] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0028] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0030] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as the target image set determined based on the image to be queried) to the terminal devices.

[0031] It should be noted that the image query method provided in this embodiment can generally be executed by server 105. Correspondingly, the image query device provided in this embodiment can generally be located in server 105. The image query method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the image query device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0032] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0033] Figure 2 This is a schematic flowchart of a method for querying images according to an embodiment of the present disclosure.

[0034] like Figure 2 As shown, the image query method 200 may include operations S210 to S230.

[0035] In operation S210, in response to receiving the image to be queried, M candidate image sets are retrieved from multiple image sets based on the image to be queried. Each of the multiple image sets includes at least one subset, and each of the at least one subset includes at least one image, where M is an integer greater than or equal to 1.

[0036] For example, cameras in the city can be used to photograph objects, obtaining images of multiple objects. Then, an image set can be created for each object and stored in a database. Alternatively, the image sets in the database can be clustered to obtain at least one subset corresponding to each image set.

[0037] For example, the image to be queried could be an image of the target object that needs to be searched.

[0038] The method for retrieving M candidate image sets from multiple image sets will be explained in detail below and will not be repeated here.

[0039] In operation S220, based on the correspondence between the image set and the subset, N subsets are determined from the M candidate image sets, where N is an integer greater than or equal to M.

[0040] For example, the correspondence between a set of images and a subset can be expressed as: the set of images includes the subset.

[0041] For example, if there are M candidate image sets including set A1 and set A2, set A1 includes subsets a11 and a12, and set A2 includes subsets a21, a22 and a23, then there are N subsets including subsets a11, a12, a21, a22 and a23.

[0042] In operation S230, a target subset is determined from the N subsets based on at least one of the first similarity and the second similarity corresponding to each of the N subsets, and is used as the target image set corresponding to the image to be queried.

[0043] For example, the first similarity is the similarity between each of the N subsets and the image to be queried. The subset with the highest first similarity among the N subsets can be determined as the reference subset. The second similarity is the similarity between each of the N subsets and the reference subset.

[0044] For example, a subset with a first similarity greater than or equal to a first threshold can be determined as the target subset, or a subset with a second similarity greater than or equal to a second threshold can be determined as the target subset, or a subset with both a first similarity greater than or equal to the first threshold and a second similarity greater than or equal to the second threshold can be determined as the target subset.

[0045] According to the technical solution provided in this disclosure, firstly, M candidate image sets are recalled; then, N subsets are determined from the M candidate image sets; and finally, a target image set is determined based on at least one of a first similarity and a second similarity corresponding to each of the N subsets. In determining the target image set, an image set can be split into at least one subset, and the candidate sets can be merged and split based on the first similarity and / or the second similarity. The process of splitting the candidate sets may include filtering out a portion of the M candidate sets, for example, filtering out subsets of other objects mixed in with the candidate sets. The process of merging the candidate sets may include merging the remaining subsets that were not filtered out. The target image set obtained through splitting and merging can improve the accuracy of image query results.

[0046] According to another embodiment of this disclosure, a database may be constructed before recalling M candidate image sets from multiple image sets.

[0047] Figure 3 This is a schematic diagram illustrating the construction of an image set according to an embodiment of the present disclosure.

[0048] like Figure 3 As shown, multiple cameras 310 in the city can be used to capture videos 320, resulting in multiple videos 320. Images can then be extracted from these videos 320; for example, images including objects can be extracted from the videos 320 according to their temporal sequence, resulting in multiple images 330. These images 330 can then be clustered to obtain multiple image sets, such as image sets 341, 342, 343, and 344, each corresponding to a single object.

[0049] Figure 4 This is a schematic diagram illustrating the construction of a database using an image set according to an embodiment of this disclosure.

[0050] The database may include an archive 431 and a vector index 432. Multiple sets of images 330 can be stored in the archive 431, and SQL (Structured Query Language) can be used as the archive 431.

[0051] A machine learning clustering algorithm can be used to perform secondary clustering on the image set 410 corresponding to each object, obtaining at least one subset corresponding to each image set 410. The machine learning clustering algorithm could be, for example, hierarchical clustering. After secondary clustering, the centroid of each of the at least one subset can be determined, and this centroid can be used to represent the image set 410. The cluster centroids can be stored in a vector index library 432.

[0052] For example, a set of images 410 contains 5 images. After secondary clustering, images pic1 and pic2 are clustered into the first subset 421, images pic3 and pic4 are clustered into the second subset 422, and image pic5 is clustered into the third subset 423.

[0053] Furthermore, after secondary clustering, the correspondence can be stored in the archive 431. The correspondence can include the correspondence between the image set 410 and its subsets, or the correspondence between the subsets and the images.

[0054] In some embodiments, the database may include a full database and a temporary database. For example, a full database may include a full archive and a full vector index, and a temporary database may include a temporary archive and a temporary vector index.

[0055] Newly generated images can be stored in the database at predetermined intervals, such as 24 hours. For cities with large populations, the large number of images generated will require a longer time to merge them into the full database.

[0056] After configuring the temporary database, you can cluster incremental images at predetermined intervals and store the clustered images in the temporary database. Then, you can initiate a scheduled task, for example, to merge the images from the temporary database into the main database starting at midnight every day. After the images in the temporary database have been merged, you can delete the temporary database.

[0057] During the process of recalling a set of M candidate images, the images to be queried can be used to recall images in both the full database and the temporary database. Furthermore, since duplicate images may exist in the full database and the temporary database before the images in the temporary database are merged, the recalled images can be deduplicated.

[0058] According to another embodiment of this disclosure, the operation of recalling M candidate image sets from multiple image sets based on the image to be queried may include the following operations: determining the centroids of each of the multiple image sets based on the images in the image sets. For example, for each image set, determining the features of each of the multiple images in the image set to obtain multiple image features, and then using the average value of the multiple image features as the centroid. Then, calculating the similarity between the image sets and the image to be queried, and recalling the image sets with similarity greater than a threshold from the multiple image sets.

[0059] It's important to note that multiple images within the same image set are inherently different. For example, some images might show the subject wearing a mask, some might show the subject's profile, and others might show the subject looking down. Therefore, the image vectors within the same set will differ from one another. Assigning a centroid to each image within the set would result in the centroid failing to represent the entire set.

[0060] Figure 5 This is a schematic flowchart of a recall operation according to an embodiment of the present disclosure.

[0061] According to another embodiment of this disclosure, the method 510 for recalling M candidate image sets from multiple image sets based on the image to be queried may include operations S511 to S513.

[0062] In operation S511, the first similarity between each subset of multiple image sets and the image to be queried is determined, resulting in multiple first similarity scores.

[0063] For example, for each image set, the images in the image set can be clustered twice to obtain at least one subset, and each subset can correspond to a cluster centroid.

[0064] For example, a database stores multiple image sets, each of which can include at least one subset; therefore, the database contains multiple subsets. The first similarity between each of these subsets and the query image can be calculated. For instance, the similarity between the cluster centroid of a subset and the query image can be used as the first similarity, resulting in multiple first similarities.

[0065] In operation S512, at least one reference subset is determined based on multiple first similarities.

[0066] For example, a recall threshold can be preset, and then the subset with the first similarity greater than or equal to the recall threshold can be determined as the reference subset.

[0067] In operation S513, based on the correspondence between the image set and the subset, the image set including at least one reference subset is determined as the candidate image set, resulting in M ​​candidate image sets.

[0068] For example, if 10 subsets are recalled and these 10 subsets are distributed across 5 image sets, then these 5 image sets are determined as the candidate image set.

[0069] The technical solution provided in this disclosure can cluster images captured by a camera device to obtain multiple image sets. Then, each of the multiple image sets is further clustered, and the image set is represented using at least one cluster centroid, thereby improving the feature representation of the image sets. Compared to the scheme of representing image sets using image set centroids, cluster centroids can more accurately represent image sets. Therefore, this disclosure can improve the recall rate of candidate image sets, and thus improve the accuracy of target image sets.

[0070] Figure 6 This is a schematic flowchart illustrating the determination of a target image set according to an embodiment of the present disclosure.

[0071] like Figure 6 As shown in this embodiment of the disclosure, the method 630 for determining a target image set from N subsets based on at least one of a first similarity and a second similarity corresponding to each of the N subsets may include operations S6301 to S6317.

[0072] In operation S6301, the evaluation value of each of the M candidate image sets is determined.

[0073] In the recalled M candidate image sets, each candidate image set includes at least one subset, and each subset corresponds to a cluster centroid. For each candidate image set, the first similarity between the cluster centroid of each at least one subset and the query image can be calculated, thus obtaining at least one first similarity corresponding to each candidate image set.

[0074] For example, the highest first similarity among at least one first similarity can be used as the evaluation value of the candidate image set. Alternatively, the weighted average of at least one first similarity can be used as the evaluation value of the candidate image set.

[0075] In operation S6302, the first category image set and / or the second category image set are determined from the M candidate image sets based on the evaluation values.

[0076] For example, the set of candidate images with the highest evaluation value among M candidate image sets can be defined as the first category image set. The other candidate image sets among the M candidate image sets besides the set with the highest evaluation value can be defined as the second category image set. It is understood that when M=1, the M candidate image sets include only one first category image set and do not include the second category image set.

[0077] In operation S6303, determine whether the current candidate image set is the first category image set. If yes, then execute operation S6304; otherwise, execute operation S6314.

[0078] In operation S6304, determine whether a rejection filtering instruction has been detected. If yes, operation S6305 can be executed; if no, it means that an acceptance filtering instruction has been detected, and operation S6306 can be executed.

[0079] For example, the reject filter instruction and the accept filter instruction are used to control whether to filter images in the first category of the image set.

[0080] Reject and accept filter commands can be configured according to actual needs. For example, when a more comprehensive set of images is needed, images of other objects can be mixed into the target image set to avoid omissions; in this case, a reject filter command can be configured. Similarly, when a more accurate selection of images is required from the target image set, an accept filter command can be configured. These commands allow users to filter images according to their specific needs, increasing the flexibility of image selection.

[0081] In operation S6305, images from the first category image set are added to the target image set.

[0082] In operation S6306, determine whether a subset of the first category of images satisfies a first predetermined condition. If yes, proceed to operation S6307; otherwise, proceed to operation S6308.

[0083] For example, the first predetermined condition may include: a first similarity to the subset is greater than or equal to a first threshold, or a second similarity to the subset is greater than or equal to a second threshold.

[0084] In operation S6307, images from the subset are added to the target image set.

[0085] It should be noted that meeting the first predetermined condition means that the subset has a high similarity to the image to be queried or the reference subset, and the images in the subset are more likely to correspond to the target object to be queried, thus ensuring the accuracy of the target image set.

[0086] In operation S6308, images from the subset are added to the set of images to be deleted.

[0087] For example, if the first category of images includes 5 subsets, and 3 of the subsets do not meet the first predetermined condition, then the images in those 3 subsets are added to the image set to be deleted.

[0088] In step S6309, determine whether a delete command has been detected. If yes, proceed to step S6310; otherwise, proceed to step S6311.

[0089] For example, similar to the reject filter command and accept filter command, the delete command can also be configured according to actual needs.

[0090] In operation S6310, images are deleted from the image set to be deleted. It can be seen that after this operation, the images from the image set to be deleted are not added to the target image set.

[0091] In operation S6311, determine whether the set of images to be deleted meets the second predetermined condition. If yes, then execute operation S6312; otherwise, execute operation S6313.

[0092] For example, the second predetermined condition includes: the number of images in the image set to be deleted is less than or equal to the first predetermined number, or the ratio of the number of images in the image set to be deleted to the number of images in the first category image set is less than or equal to the first predetermined ratio.

[0093] For example, if the first category of images contains 100 images and the set of images to be deleted contains 10 images, then the ratio of the number of images in the set of images to be deleted to the number of images in the first category of images is 0.1. In this embodiment, the first predetermined quantity is set to 20, and the first predetermined ratio is set to 0.2. It can be seen that the set of images to be deleted satisfies the second predetermined condition.

[0094] In operation S6312, images from the image set to be deleted are added to the target image set.

[0095] It should be noted that if the set of images to be deleted meets the second predetermined condition, it means that the images in the set of images to be deleted have been misjudged, and therefore the images in the set of images to be deleted can be retained.

[0096] In operation S6313, delete the images in the image collection to be deleted.

[0097] It should be noted that if the set of images to be deleted does not meet the second predetermined condition, it indicates that the quality of the candidate image set corresponding to that set is low, and that some images in the candidate image set are incorrect. For example, the candidate image set corresponds to object A, while the incorrect images correspond to object B. Therefore, the images in the set of images to be deleted can be deleted.

[0098] In operation S6314, for the second category of images in the M candidate image sets, the target set is determined from the second category of images based on at least one of the first similarity and the second similarity corresponding to each of the N subsets.

[0099] For example, for each second-category image set in M ​​candidate image sets, a candidate subset satisfying a fourth predetermined condition can be determined within each second-category image set. Then, if each second-category image set satisfies a fifth predetermined condition, each second-category image set is determined as the target set.

[0100] For example, the fourth predetermined condition includes: the first similarity to the subset is greater than or equal to the fifth threshold, and the second similarity to the subset is greater than or equal to the sixth threshold;

[0101] For example, the fifth predetermined condition includes: the number of images in each second category image set is greater than or equal to a second predetermined quantity, and the number of candidate subsets is greater than or equal to a third predetermined quantity, and the ratio of the number of candidate subsets to the number of subsets in each second category image set is greater than or equal to a second predetermined ratio. For example, the second predetermined quantity could be 3.

[0102] For example, the fifth predetermined condition includes: the number of images in each second category image set is greater than or equal to the second predetermined number, and the number of images in the candidate subset is greater than or equal to the fourth predetermined number, and the ratio of the number of images in the candidate subset to the number of images in each second category image set is greater than or equal to the third predetermined ratio.

[0103] It should be noted that if the number of images in the second category image set is greater than or equal to the second predetermined number, then multiple images in the second category image set can be used to represent the characteristics of the object.

[0104] If the number of images in the second category image set is less than the second predetermined number, it indicates that the second category image set contains fewer images, and there is a risk that the images may be incorrect. For example, the image to be queried is an image of object A wearing a mask. However, a certain second category image set contains only one image, which is an image of object B wearing a mask. Because the mask covers the face, a subset may meet the third predetermined condition, but object A and object B are different objects. It can be seen that because the number of images in the second category image set is small, it is impossible to accurately determine whether the object corresponding to the second category image set is the same as the object in the image to be queried.

[0105] It can be seen that by using the fourth and fifth pre-defined conditions, a high-quality target set can be selected, thereby ensuring the accuracy of the target image set during the process of selecting images from the candidate target set.

[0106] In operation S6315, determine whether a rejection filtering instruction has been detected. If yes, proceed to operation S6316; otherwise, proceed to operation S6317.

[0107] In operation S6316, images from the target set are added to the target image set.

[0108] For example, the deny filter directive controls whether images in a target image set are filtered. The deny filter directive can be configured according to actual needs. It allows users to filter images based on their specific requirements, increasing the flexibility of image selection.

[0109] In operation S6317, images from the subset of the target set that meet the third predetermined condition are added to the target image set.

[0110] For example, the third predetermined condition includes: the first similarity to the subset is greater than or equal to the third threshold.

[0111] For example, the third predefined condition includes: the second similarity to the subset is greater than or equal to the fourth threshold. In practical applications, when the image to be queried is of poor quality due to poor lighting, shooting angle, or blurry image, the second similarity between the subset and the reference subset can be used as a factor in the third predefined condition, thereby improving the accuracy of the target image set.

[0112] It should be noted that in the above embodiments, the threshold, predetermined quantity, and predetermined ratio can all be set according to actual needs. For example, the threshold can be 0.6 to 0.8, and the threshold can be any one of the first threshold, second threshold, third threshold, fourth threshold, fifth threshold, and sixth threshold. For example, the predetermined quantity can be 3, 10, 30, 50, etc., and the predetermined quantity can be any one of the first predetermined quantity, second predetermined quantity, and third predetermined quantity. The predetermined ratio can be 0.2, 0.3, 0.7, etc., and the predetermined ratio can be any one of the first predetermined ratio, second predetermined ratio, and second predetermined ratio.

[0113] It should be noted that in the above embodiment, operation S6302 divides the M candidate image sets into a first category image set and a second category image set. In other embodiments, the categories of the candidate image sets may not be distinguished. For example, all M candidate image sets may be determined as the first category image set. Or, for another example, all M candidate image sets may be determined as the second category image set.

[0114] It should be noted that in some embodiments, the above operations S6304 and S6305 can be omitted, that is, after performing operation S6303, operation S6306 is performed on the first category of image set.

[0115] It should be noted that in some embodiments, the above operations S6309 and S6310 can be omitted, that is, operation S6311 is performed after operation S6308.

[0116] According to another embodiment of this disclosure, after obtaining the target image set, the target image set can be cached. The target image set can be returned to the client and displayed using the client. Furthermore, an evaluation value for the target image set can be calculated and displayed using the client. For example, if the target image set includes multiple subsets, multiple first similarities between each subset and the image to be queried can be determined, and then the median of the multiple first similarities is used as the evaluation value of the target image set.

[0117] Figure 7 This is a schematic diagram illustrating the method for querying images according to an embodiment of this disclosure.

[0118] like Figure 7 As shown, databases can be pre-built, for example, a full database 703 and a temporary database 704.

[0119] After receiving the image to be queried 701, features 702 of the image to be queried can be extracted. Then, using the features 702, M candidate image sets 705 can be recalled from multiple image sets. For example, recall operations can be performed from the full database 703 and the temporary database 704 respectively. Then, the recalled images are deduplicated to obtain M candidate image sets 705.

[0120] Then, based on the correspondence between the image set and the subset, N subsets 706 can be determined from the M candidate image sets 705.

[0121] Then, a target subset can be determined from the N subsets 706 as the target image set 707 corresponding to the image to be queried 701. For example, based on at least one of the first similarity and second similarity corresponding to each of the N subsets 706, a subset of the N subsets 706 is filtered, another subset is determined as the target subset, and the images in the target subset are added to the target image set 707.

[0122] The target image set 707 can then be returned to the client 708 and displayed using the client 708.

[0123] Figure 8 This is a schematic structural block diagram of a device for querying images according to an embodiment of the present disclosure.

[0124] like Figure 8 As shown, the image query device 800 may include a recall module 810, a first determination module 820, and a second determination module 830.

[0125] The recall module 810 is used to respond to receiving a query image and, based on the query image, recall M candidate image sets from multiple image sets. Each of the multiple image sets includes at least one subset, and each of the at least one subset includes at least one image; M is an integer greater than or equal to 1.

[0126] The first determining module 820 is used to determine N subsets from M candidate image sets based on the correspondence between the image set and the subsets, where N is an integer greater than or equal to M.

[0127] The second determining module 830 is used to determine a target subset from the N subsets based on at least one of a first similarity and a second similarity corresponding to each of the N subsets, as the target image set corresponding to the image to be queried; wherein, the first similarity is the similarity between each of the N subsets and the image to be queried, the second similarity is the similarity between each of the N subsets and a reference subset, and the reference subset is the subset among the N subsets that corresponds to the largest first similarity.

[0128] According to another embodiment of this disclosure, the second determining module includes a first determining submodule, configured to, for a first category of image sets among M candidate image sets, add images from a subset of the first category of image sets to a target image set if a subset satisfies a first predetermined condition; wherein the first category of image sets is the candidate image set with the highest evaluation value among the M candidate image sets; the first predetermined condition includes: a first similarity to the subset being greater than or equal to a first threshold, or a second similarity to the subset being greater than or equal to a second threshold.

[0129] According to another embodiment of this disclosure, the second determining module includes a second determining submodule, configured to, for a first category of image sets in M ​​candidate image sets, add images from a subset of the first category of image sets to a set to be deleted if a subset of the first category of image sets does not meet a first predetermined condition; and add images from the set to be deleted to a target image set if the set to be deleted meets a second predetermined condition; wherein the first category of image sets is the candidate image set with the highest evaluation value among the M candidate image sets; the first predetermined condition includes: a first similarity to a subset greater than or equal to a first threshold, or a second similarity to a subset greater than or equal to a second threshold; the second predetermined condition includes: the number of images in the set to be deleted is less than or equal to a first predetermined number, or the ratio of the number of images in the set to be deleted to the number of images in the first category of image sets is less than or equal to a first predetermined ratio.

[0130] According to another embodiment of this disclosure, the second determining module includes a third determining submodule, configured to, for a second category of image sets in a set of M candidate images, determine a target set from the set of M candidate images based on at least one of a first similarity and a second similarity corresponding to each of the N subsets; add images from the target set to the target image set in response to detecting a rejection filtering instruction; and add images from the subsets of the target set that satisfy a third predetermined condition to the target image set in response to detecting an acceptance filtering instruction; wherein the second category of image sets are the other candidate image sets in the set of M candidate images excluding the candidate image set with the highest evaluation value; the third predetermined condition includes: a first similarity corresponding to a subset greater than or equal to a third threshold, or a second similarity corresponding to a subset greater than or equal to a fourth threshold.

[0131] According to another embodiment of this disclosure, the third determining submodule includes a determining unit, configured to, for each second category image set in M ​​candidate image sets, determine a candidate subset in each second category image set that satisfies a fourth predetermined condition; and, if it is determined that each second category image set satisfies a fifth predetermined condition, determine each second category image set as a target set; wherein the fourth predetermined condition includes: a first similarity corresponding to the subset is greater than or equal to a fifth threshold, and a second similarity corresponding to the subset is greater than or equal to a sixth threshold; the fifth predetermined condition includes: the number of images in each second category image set is greater than or equal to a second predetermined number, and the number of candidate subsets is greater than or equal to a third predetermined number, and the ratio of the number of candidate subsets to the number of subsets in each second category image set is greater than or equal to a second predetermined ratio; or the number of images in each second category image set is greater than or equal to a second predetermined number, and the number of images in the candidate subsets is greater than or equal to a fourth predetermined number, and the ratio of the number of images in the candidate subsets to the number of images in each second category image set is greater than or equal to a third predetermined ratio.

[0132] According to another embodiment of this disclosure, the above-described image querying apparatus further includes an adding module and a triggering module. The adding module is configured to add images from the first category image set to the target image set in response to detecting a rejection filtering instruction. The triggering module is configured to trigger an operation to determine the target subset in response to detecting an acceptance filtering instruction.

[0133] According to another embodiment of this disclosure, the recall module includes a first similarity determination submodule, a reference subset determination submodule, and a candidate image set determination submodule. The first similarity determination submodule is used to determine a first similarity between each of multiple subsets in a plurality of image sets and the image to be queried, thus obtaining multiple first similarities. The reference subset determination submodule is used to determine at least one reference subset based on the multiple first similarities. The candidate image set determination submodule is used to determine the image set, including at least one reference subset, as a candidate image set based on the correspondence between image sets and subsets, thus obtaining M candidate image sets.

[0134] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0135] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0136] According to embodiments of this disclosure, this disclosure also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method for querying images.

[0137] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the above-described method for querying images.

[0138] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for querying images.

[0139] Figure 9 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0140] like Figure 9As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0141] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0142] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the method of querying an image. For example, in some embodiments, the method of querying an image may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the method of querying an image described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to perform the method of querying an image by any other suitable means (e.g., by means of firmware).

[0143] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0147] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0148] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0149] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0150] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for querying a picture, comprising: in response to receiving a picture to be queried, recalling, from a plurality of picture sets, M candidate picture sets according to the picture to be queried, the plurality of picture sets being obtained by clustering pictures, the plurality of picture sets each comprising at least one sub-set, each of the at least one sub-set corresponding to a cluster center, and each of the at least one sub-set comprising at least one picture, M being an integer greater than or equal to 1; determining, according to a correspondence between the picture sets and the sub-sets, N sub-sets included in the M candidate picture sets, N being an integer greater than or equal to M; and determining, from the N sub-sets, a target sub-set as a target picture set corresponding to the picture to be queried, according to at least one of a first similarity and a second similarity corresponding to each of the N sub-sets; wherein the first similarity is a similarity between a cluster center corresponding to each of the N sub-sets and the picture to be queried, and the second similarity is a similarity between each of the N sub-sets and a reference sub-set, the reference sub-set being a sub-set corresponding to a maximum first similarity among the N sub-sets.

2. The method of claim 1, wherein, The determining, from the N sub-sets, a target sub-set as a target picture set corresponding to the picture to be queried, according to at least one of a first similarity and a second similarity corresponding to each of the N sub-sets, comprises: for a first type of picture set among the M candidate picture sets: in a case where it is determined that a sub-set in the first type of picture set satisfies a first predetermined condition, adding pictures in the sub-set to the target picture set; wherein the first type of picture set is a candidate picture set with a highest evaluation value among the M candidate picture sets; the first predetermined condition comprises that the first similarity corresponding to the sub-set is greater than or equal to a first threshold value, or the second similarity corresponding to the sub-set is greater than or equal to a second threshold value.

3. The method of claim 1, wherein, The determining, from the N sub-sets, a target sub-set as a target picture set corresponding to the picture to be queried, according to at least one of a first similarity and a second similarity corresponding to each of the N sub-sets, comprises: for a first type of picture set among the M candidate picture sets: in a case where it is determined that a sub-set in the first type of picture set does not satisfy a first predetermined condition, adding pictures in the sub-set to a picture set to be deleted; and in a case where it is determined that the picture set to be deleted satisfies a second predetermined condition, adding pictures in the picture set to be deleted to the target picture set; wherein the first type of picture set is a candidate picture set with a highest evaluation value among the M candidate picture sets; the first predetermined condition comprises that the first similarity corresponding to the sub-set is greater than or equal to a first threshold value, or the second similarity corresponding to the sub-set is greater than or equal to a second threshold value. The second predetermined condition comprises: a number of pictures in the picture set to be deleted is less than or equal to a first predetermined number, or a ratio of the number of pictures in the picture set to be deleted to a number of pictures in the first category picture set is less than or equal to a first predetermined ratio.

4. The method of claim 1, wherein, The determining the target picture set corresponding to the picture to be queried from the N sub-sets according to at least one of the first similarity and the second similarity corresponding to each of the N sub-sets comprises: For a second category picture set in the M candidate picture sets: Determining a target set from the second category picture set according to at least one of the first similarity and the second similarity corresponding to each of the N sub-sets; In response to detecting the rejection filtering instruction, adding pictures in the target set to the target picture set; and In response to detecting the acceptance filtering instruction, adding pictures in a sub-set in the target set that satisfies a third predetermined condition to the target picture set. The second category picture set is a candidate picture set in the M candidate picture sets other than the candidate picture set with the highest evaluation value. The third predetermined condition comprises: the first similarity corresponding to the sub-set is greater than or equal to a third threshold value, or the second similarity corresponding to the sub-set is greater than or equal to a fourth threshold value.

5. The method of claim 4, wherein, The determining the target picture set corresponding to the picture to be queried from the N sub-sets according to at least one of the first similarity and the second similarity corresponding to each of the N sub-sets comprises: For each second category picture set in the M candidate picture sets: Determining a candidate sub-set in the each second category picture set that satisfies a fourth predetermined condition; and In a case where it is determined that the each second category picture set satisfies a fifth predetermined condition, determining the each second category picture set as the target set. The fourth predetermined condition comprises: the first similarity corresponding to a sub-set is greater than or equal to a fifth threshold value, and the second similarity corresponding to the sub-set is greater than or equal to a sixth threshold value. The fifth predetermined condition comprises: A number of pictures in the each second category picture set is greater than or equal to a second predetermined number, and a number of the candidate sub-sets is greater than or equal to a third predetermined number, and a ratio of the number of the candidate sub-sets to a number of sub-sets in the each second category picture set is greater than or equal to a second predetermined ratio; or A number of pictures in the each second category picture set is greater than or equal to a second predetermined number, and a number of pictures in the candidate sub-set is greater than or equal to a fourth predetermined number, and a ratio of the number of pictures in the candidate sub-set to the number of pictures in the each second category picture set is greater than or equal to a third predetermined ratio.

6. The method of claim 2 or 3, further comprising: In response to detecting the rejection filtering instruction, adding pictures in the first category picture set to the target picture set; and In response to detecting the acceptance filtering instruction, triggering the operation of determining the target sub-set. ​ 7. The method of claim 1, wherein, The recalling, in response to receiving the to-be-searched picture, M candidate picture sets from a plurality of picture sets according to the to-be-searched picture comprises: determining a first similarity between each of a plurality of sub-sets in the plurality of picture sets and the to-be-searched picture, to obtain a plurality of first similarities; determining at least one reference sub-set according to the plurality of first similarities; and determining, according to a corresponding relationship between picture sets and sub-sets, picture sets including the at least one reference sub-set as candidate picture sets, to obtain the M candidate picture sets.

8. An apparatus for searching a picture, comprising: a recalling module configured to, in response to receiving a to-be-searched picture, recall M candidate picture sets from a plurality of picture sets according to the to-be-searched picture, the plurality of picture sets being obtained by clustering pictures, the plurality of picture sets each including at least one sub-set by performing secondary clustering on the plurality of picture sets respectively, the at least one sub-set each corresponding to a cluster center, and the at least one sub-set each including at least one picture; M being an integer greater than or equal to 1; a first determining module configured to determine N sub-sets included in the M candidate picture sets according to a corresponding relationship between picture sets and sub-sets, N being an integer greater than or equal to M; and a second determining module configured to determine a target sub-set from the N sub-sets as a target picture set corresponding to the to-be-searched picture according to at least one of a first similarity and a second similarity corresponding to the N sub-sets respectively; wherein the first similarity is a similarity between a cluster center corresponding to the N sub-sets respectively and the to-be-searched picture, and the second similarity is a similarity between the N sub-sets respectively and a reference sub-set, the reference sub-set being a sub-set corresponding to a maximum first similarity in the N sub-sets. The second determining module comprises:

9. The apparatus of claim 8, wherein, a first determining sub-module configured to, for a first type picture set in the M candidate picture sets: in a case where it is determined that a sub-set in the first type picture set satisfies a first predetermined condition, add pictures in the sub-set to the target picture set; wherein the first type picture set is a candidate picture set with a highest evaluation value in the M candidate picture sets; the first predetermined condition comprises that the first similarity corresponding to the sub-set is greater than or equal to a first threshold value, or the second similarity corresponding to the sub-set is greater than or equal to a second threshold value. The second determining module comprises:

10. The apparatus of claim 8, wherein, a second determining sub-module configured to, for a first type picture set in the M candidate picture sets: in a case where it is determined that a sub-set in the first type picture set does not satisfy a first predetermined condition, add pictures in the sub-set to a to-be-deleted picture set; and in a case where it is determined that the to-be-deleted picture set satisfies a second predetermined condition, add pictures in the to-be-deleted picture set to the target picture set; wherein the first type picture set is a candidate picture set with a highest evaluation value in the M candidate picture sets. ​ The first predetermined condition comprises that the first similarity corresponding to the sub-set is greater than or equal to a first threshold value, or the second similarity corresponding to the sub-set is greater than or equal to a second threshold value. The second predetermined condition comprises that the number of pictures in the picture set to be deleted is less than or equal to a first predetermined number, or the ratio of the number of pictures in the picture set to be deleted to the number of pictures in the first category picture set is less than or equal to a first predetermined ratio.

11. The apparatus of claim 8, wherein, The second determining module comprises: A third determining submodule is configured to determine, for a second category picture set in the M candidate picture sets: determine a target set from the second category picture set according to at least one of the first similarity and the second similarity corresponding to each of the N sub-sets; add the pictures in the target set to the target picture set in response to detecting a rejection filtering instruction; and add the pictures in a sub-set in the target set that meets a third predetermined condition to the target picture set in response to detecting an acceptance filtering instruction. The second category picture set is a candidate picture set other than the candidate picture set with the highest evaluation value in the M candidate picture sets. The third predetermined condition comprises that the first similarity corresponding to the sub-set is greater than or equal to a third threshold value, or the second similarity corresponding to the sub-set is greater than or equal to a fourth threshold value.

12. The apparatus of claim 11, wherein, The third determining submodule comprises: A determining unit is configured to determine, for each second category picture set in the M candidate picture sets: determine a candidate sub-set in the each second category picture set that meets a fourth predetermined condition; and determine the each second category picture set as the target set in a case where the each second category picture set meets a fifth predetermined condition. The fourth predetermined condition comprises that the first similarity corresponding to a sub-set is greater than or equal to a fifth threshold value, and the second similarity corresponding to the sub-set is greater than or equal to a sixth threshold value. The fifth predetermined condition comprises that the number of pictures in the each second category picture set is greater than or equal to a second predetermined number, the number of candidate sub-sets is greater than or equal to a third predetermined number, and the ratio of the number of candidate sub-sets to the number of sub-sets in the each second category picture set is greater than or equal to a second predetermined ratio; or The number of pictures in the each second category picture set is greater than or equal to a second predetermined number, the number of pictures in the candidate sub-set is greater than or equal to a fourth predetermined number, and the ratio of the number of pictures in the candidate sub-set to the number of pictures in the each second category picture set is greater than or equal to a third predetermined ratio.

13. The apparatus of claim 9 or 10, further comprising: an adding module configured to add the pictures in the first category picture set to the target picture set in response to detecting a rejection filtering instruction; and a triggering module configured to trigger an operation of determining a target sub-set in response to detecting an acceptance filtering instruction. The recall module comprises:

14. The apparatus of claim 8, wherein, ​ a first similarity determination submodule, configured to determine a first similarity between each of a plurality of sub-sets of the plurality of picture sets and the picture to be queried, to obtain a plurality of first similarities; a reference sub-set determination submodule, configured to determine at least one reference sub-set according to the plurality of first similarities; and a candidate picture set determination submodule, configured to determine, according to a corresponding relationship between a picture set and a sub-set, a picture set including the at least one reference sub-set as a candidate picture set, to obtain the M candidate picture sets.

15. An electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1 to 7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 7.

Images