Similar image determination method and device, electronic equipment and program product

Abstract

The invention relates to a similar image determination method and device, electronic equipment and a computer program product, relates to the technical field of computer vision, and can be applied to similar image retrieval scenes. The method comprises the following steps: acquiring to-be-retrieved information, and performing feature extraction processing on the to-be-retrieved information to obtain to-be-retrieved features of the to-be-retrieved information; obtaining a pre-configured clustering feature library, wherein the clustering feature library comprises category image features corresponding to the plurality of image categories; determining a similar image category matched with the to-be-retrieved feature according to the plurality of category image features; and according to the similar image category, determining a similar image corresponding to the to-be-retrieved information. According to the invention, the calculation amount can be saved in similar image query operation, and the query efficiency is improved.

Description

Background Art

[0002] Image search refers to searching for images similar to the given query image in the image library, and text search refers to searching for images similar to the given text description in the image library. In the image search and text search tasks, usually the input query image or query text is received, and image query is performed in the retrieval database based on the query image or query text. During the query process, it is necessary to calculate the similar image sequence that is most similar to the query image in the retrieval image library, and then output the similar image sequence.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a similar image determination method, a similar image determination device, an electronic device, a computer-readable storage medium and a computer program product, thereby at least to a certain extent overcoming the problem that in related image retrieval tasks, as the number of retrieved images increases, the amount of query calculation also increases synchronously, resulting in each query taking a long time and reducing the image query efficiency.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present invention.

[0006] According to a first aspect of the present disclosure, a similar image determination method is provided, comprising: obtaining information to be retrieved, performing feature extraction processing on the information to be retrieved, and obtaining features to be retrieved of the information to be retrieved; obtaining a preconfigured cluster feature library, the cluster feature library comprising category image features corresponding to a plurality of image categories; determining a similar image category matching the features to be retrieved based on the plurality of category image features; and determining a similar image corresponding to the information to be retrieved based on the similar image category.

[0007] In an exemplary embodiment of the present disclosure, the feature extraction processing of the information to be retrieved to obtain the features to be retrieved of the information to be retrieved includes: determining image retrieval task information corresponding to the information to be retrieved; and based on the image retrieval task information, performing feature extraction processing on the information to be retrieved to obtain the features to be retrieved.

[0008] In an exemplary embodiment of the present disclosure, based on the image retrieval task information, feature extraction processing is performed on the information to be retrieved to obtain the features to be retrieved, including: when the image retrieval task information is to retrieve images with the same semantics, image feature extraction and text feature extraction are performed on the information to be retrieved to obtain the features to be retrieved; when the image retrieval task information is to retrieve images with similar pixels, structural feature extraction and texture feature extraction are performed on the information to be retrieved to obtain the features to be retrieved; when the image retrieval task information is to retrieve images with similar contents, image vector feature extraction is performed on the information to be retrieved to obtain the features to be retrieved.

[0009] In an exemplary embodiment of the present disclosure, the obtaining of a preconfigured clustering feature library includes: obtaining a pre-constructed image database, the image database including a plurality of images; performing feature extraction processing on the plurality of images based on pre-configured feature categories to construct a plurality of feature databases, the feature categories including one or more of image pixel features, image semantic features and image content features; performing feature clustering processing on the plurality of images in each of the feature databases to obtain a clustering feature library.

[0010] In an exemplary embodiment of the present disclosure, the cluster feature library includes multiple image categories, and the feature clustering processing is performed on each of the multiple images in the feature database to obtain the cluster feature library, including: determining the feature distance between different images in the feature database; performing feature clustering processing on the multiple images according to the feature distance to obtain multiple image categories with different image features, and the feature distance between any two images in the image category is less than a distance threshold.

[0011] In an exemplary embodiment of the present disclosure, determining the similar image category corresponding to the feature to be retrieved based on the multiple category image features includes: determining a first feature distance between the feature to be retrieved and each of the category image features; selecting similar image features from the category image features based on the first feature distance; and taking the image category in which the matching similar image features are located as the similar image category.

[0012] In an exemplary embodiment of the present disclosure, determining similar images corresponding to the information to be retrieved based on the similar image categories includes: obtaining candidate image features corresponding to each image under each of the similar image categories; determining a second feature distance between the feature to be retrieved and the candidate image feature; and generating a similar image list based on a distance value of the second feature distance, wherein the similar image list includes one or more of the similar images.

[0013] According to a second aspect of the present disclosure, a similar image determination device is provided, comprising: a retrieval information acquisition module, used to acquire information to be retrieved, perform feature extraction processing on the information to be retrieved, and obtain features to be retrieved of the information to be retrieved; a feature library acquisition module, used to acquire a pre-configured cluster feature library, the cluster feature library including category image features corresponding to multiple image categories; a similar category determination module, used to determine a similar image category matching the features to be retrieved based on the multiple category image features; and a similar image determination module, used to determine similar images corresponding to the information to be retrieved based on the similar image category.

[0014] In an exemplary embodiment of the present disclosure, the retrieval information acquisition module includes a retrieval feature extraction unit, which is used to: determine the image retrieval task information corresponding to the information to be retrieved; based on the image retrieval task information, perform feature extraction processing on the information to be retrieved to obtain the features to be retrieved.

[0015] In an exemplary embodiment of the present disclosure, the retrieval feature extraction unit includes a retrieval feature extraction sub-unit, which is used to: when the image retrieval task information is to retrieve images with the same semantics, perform image feature extraction and text feature extraction on the information to be retrieved to obtain the features to be retrieved; when the image retrieval task information is to retrieve images with similar pixels, perform structural feature extraction and texture feature extraction on the information to be retrieved to obtain the features to be retrieved; when the image retrieval task information is to retrieve images with similar content, perform image vector feature extraction on the information to be retrieved to obtain the features to be retrieved.

[0016] In an exemplary embodiment of the present disclosure, the feature library acquisition module also includes a feature library construction unit, which is used to: obtain a pre-constructed image database, the image database includes multiple images; based on pre-configured feature categories, perform feature extraction processing on the multiple images to construct multiple feature databases, the feature categories include one or more of image pixel features, image semantic features and image content features; perform feature clustering processing on each of the multiple images in the feature database to obtain a clustered feature library.

[0017] In an exemplary embodiment of the present disclosure, the clustering feature library includes multiple image categories, and the feature library construction unit includes a feature library construction sub-unit, which is used to: determine the feature distance between different images in the feature database; perform feature clustering processing on the multiple images according to the feature distance to obtain multiple image categories with different image features, and the feature distance between any two images in the image category is less than a distance threshold.

[0018] In an exemplary embodiment of the present disclosure, the similar category determination module includes a similar category determination unit, which is used to: determine a first feature distance between the feature to be retrieved and each of the category image features; select similar image features from the category image features based on the first feature distance; and use the image category in which the matching similar image features are located as a similar image category.

[0019] In an exemplary embodiment of the present disclosure, the similar image determination module includes a similar image determination unit, which is used to: obtain candidate image features corresponding to each image under each of the similar image categories; determine a second feature distance between the feature to be retrieved and the candidate image feature; and generate a similar image list based on the distance value of the second feature distance, wherein the similar image list contains one or more of the similar images.

[0020] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the similar image determination method according to any one of the above items is implemented.

[0021] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the similar image determination method according to any one of the above items is implemented.

[0022] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which implements any one of the above-described similar image determination methods when executed by a processor.

[0023] The technical solution provided by the present disclosure may have the following beneficial effects:

[0024] In the exemplary embodiment of the present disclosure, the similar image determination method, on the one hand, compares the features to be retrieved with the image features in the cluster feature library to determine the similar image category, and then determines the similar images from the similar image category, which can greatly reduce the amount of query calculation and improve the efficiency of image query. On the other hand, by determining similar images from similar image categories, it is only necessary to compare with several categories of features with similar features, which can reduce the feature distance calculation and the amount of image retrieval calculation. The larger the image library, the more time is saved.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0027] Figure 1 The main framework diagram of the system used in the image retrieval task in the relevant scheme is shown;

[0028] Figure 2 A flowchart of a similar image determination method according to an exemplary embodiment of the present disclosure is schematically shown;

[0029] Figure 3 Schematically shows a functional framework diagram according to an exemplary embodiment of the present disclosure that is improved compared to related solutions;

[0030] Figure 4 The overall flow chart of performing feature matching based on a cluster feature library to determine similar images according to an exemplary embodiment of the present disclosure is schematically shown;

[0031] Figure 5 A flowchart of building a clustering feature library according to an exemplary embodiment of the present disclosure is schematically shown;

[0032] Figure 6 A block diagram schematically shows a similar image determination apparatus according to an exemplary embodiment of the present disclosure;

[0033] Figure 7 A block diagram schematically shows an electronic device according to an exemplary embodiment of the present disclosure;

[0034] Figure 8 A schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0035] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0036] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0037] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0038] In the image search and text search tasks, as the number of retrieved images increases, the amount of query calculation also increases synchronously, resulting in each query taking a lot of time and reducing the efficiency of image query. Figure 1 , Figure 1 The main framework diagram of the system used in the image retrieval task in the relevant scheme is shown. In step S110, the retrieval image is input; in step S120, the feature extraction process is performed on the retrieval image to obtain the corresponding image features; then in step S130, the matching is performed in the data set feature library, and through the matching operation, in step S140, the corresponding similar image sequence is output.

[0039] In the feature extraction process, different image feature extraction methods are used for different similar image search requirements. For example, when semantic similarity occurs, an image encoder based on the contrastive language-image pre-training (CLIP) model can be used. The CLIP model is modal and includes both image processing and text processing. When the images are similar overall, a computer vision self-supervised model (DINOv2) can be used for feature extraction. The above solutions have technical problems such as large databases and long waiting times for user retrieval.

[0040] Based on this, in this example embodiment, a similar image determination method is first provided. The similar image determination method of the present disclosure can be implemented using a server, or the method described in the present disclosure can be implemented using a terminal device, wherein the terminal described in the present disclosure may include mobile terminals such as mobile phones, tablet computers, laptop computers, PDAs, and personal digital assistants (PDAs), as well as fixed terminals such as desktop computers. Figure 2 The following is a schematic diagram schematically showing the process of a similar image determination method according to some embodiments of the present disclosure. Figure 2 , the similar image determination method may include the following steps:

[0041] Step S210, obtaining the information to be retrieved, performing feature extraction processing on the information to be retrieved, and obtaining the features to be retrieved of the information to be retrieved.

[0042] According to some exemplary embodiments of the present disclosure, the information to be retrieved may be key information based on the image retrieval task. The features to be retrieved may be corresponding features obtained after feature extraction is performed on the information to be retrieved.

[0043] In the image search task, the information to be retrieved can be obtained first. The information to be retrieved can be obtained by obtaining user input or by a pre-configured image retrieval task. The information to be retrieved can be an image, such as an image search task, or a text, such as a text search task. After the information to be retrieved is obtained, feature extraction is performed on the information to obtain the corresponding features to be retrieved.

[0044] Step S220: obtaining a pre-configured clustering feature library, where the clustering feature library includes category image features corresponding to each of a plurality of image categories.

[0045] According to some exemplary embodiments of the present disclosure, the cluster feature library may be a database for storing multiple image categories, and the cluster feature library is obtained by clustering the features of multiple images in the image database. The category image features may be standard reference features corresponding to each image category in the cluster feature library.

[0046] Obtain a clustering feature library. The clustering feature library is a feature clustering result formed by clustering multiple image features in the image database. The clustering feature library can contain multiple image categories. In the clustering feature library, for each image category, the feature of a certain image (such as the first image) can usually be used as the category image feature corresponding to the image category, so as to perform subsequent feature distance calculation.

[0047] Step S230: determining similar image categories matching the features to be retrieved based on the multiple category image features.

[0048] According to some exemplary embodiments of the present disclosure, a similar image category may be an image category that has a high similarity with the feature to be retrieved. For example, when the similarity between the category image feature corresponding to a certain image category and the feature to be retrieved is within a specified numerical range, the image category may be used as a similar image category that matches the feature to be retrieved.

[0049] After extracting the features to be retrieved, a similarity comparison is performed between the features to be retrieved and the category image features corresponding to multiple image categories in the clustering feature library, and the best matching one or more similar image features are queried in the clustering feature library. For example, the two or three most similar category features can be selected as similar image features; and the image category in which the above similar image features are located is taken as the similar image category.

[0050] Step S240: determining similar images corresponding to the information to be retrieved according to the similar image categories.

[0051] According to some exemplary embodiments of the present disclosure, the similar image may be an image whose image features match the information to be retrieved.

[0052] After screening out similar image categories, similarity matching is performed based on the features to be retrieved and the images in the similar image categories. For example, full or partial matching can be performed on the images in the similar image categories to determine one or more similar images corresponding to the information to be retrieved, so that the determined multiple similar images can be subsequently displayed in a list form.

[0053] According to the similar image determination method in this example embodiment, on the one hand, for the features to be retrieved, they are first compared with the image features in the cluster feature library to determine the similar image category, and then similar images are determined from the similar image category, which can greatly reduce the amount of query calculation and improve the efficiency of image query. On the other hand, by determining similar images from similar image categories, it is only necessary to compare with several categories of features with similar features, which can reduce feature distance calculation and image retrieval calculation. The larger the image library, the more time is saved.

[0054] Next, the similar image determination method in this exemplary embodiment will be further described.

[0055] In an exemplary embodiment of the present disclosure, for step S210, feature extraction processing is performed on the information to be retrieved to obtain features to be retrieved of the information to be retrieved, including: determining image retrieval task information corresponding to the information to be retrieved; based on the image retrieval task information, feature extraction processing is performed on the information to be retrieved to obtain features to be retrieved.

[0056] The image retrieval task information may be the relevant information based on which the current image retrieval task is performed, and the image retrieval task information may include the feature category corresponding to the information to be retrieved.

[0057] refer to Figure 3 , Figure 3 The improved functional framework diagram according to the exemplary embodiment of the present disclosure compared with the related solutions is schematically shown. In step S310, the search information is input. In step S320, the search feature extraction is performed on the search information to obtain the search feature.

[0058] Since different image feature methods can be used in different similar image search requirements, a variety of different similarity query tasks are generated in the image retrieval task. The present disclosure can be applied to different similarity query tasks, such as image semantic level, pixel level similarity, etc. After obtaining the information to be retrieved, the corresponding image retrieval task information is determined, and the image retrieval task information can include the specific requirements of this image retrieval task, that is, the specific similarity category used in this image retrieval task.

[0059] After determining the image retrieval task information, feature extraction can be performed on the information to be retrieved according to the image retrieval task information to obtain the corresponding features to be retrieved. Feature extraction is performed according to the similarity category used in this image retrieval task, and then the features to be retrieved that match the information to be retrieved can be directly determined from the clustering feature library for subsequent similarity calculation.

[0060] In an exemplary embodiment of the present disclosure, based on the image retrieval task information, feature extraction processing is performed on the information to be retrieved to obtain the features to be retrieved, including: when the image retrieval task information is to retrieve images with the same semantics, image feature extraction and text feature extraction are performed on the information to be retrieved to obtain the features to be retrieved; when the image retrieval task information is to retrieve images with similar pixels, structural feature extraction and texture feature extraction are performed on the information to be retrieved to obtain the features to be retrieved; when the image retrieval task information is to retrieve images with similar contents, image vector feature extraction is performed on the information to be retrieved to obtain the features to be retrieved.

[0061] The present disclosure selects different feature extraction methods for different image similarity query tasks. Specifically, in the case of searching images with text, or searching images with the same semantics with images, the image retrieval task information corresponds to the images with the same semantics. At this time, image feature extraction and text feature extraction can be performed on the information to be retrieved to obtain the features to be retrieved; for example, the clip model can be used to extract image features and text features.

[0062] In the case of image search or image pixel-level similarity, the image retrieval task information corresponds to retrieving images with similar pixels. At this time, traditional image structure features or texture features can be used for feature extraction to obtain the features to be retrieved.

[0063] In the case of searching for images to identify similar contents, when the image retrieval task information is to retrieve images with similar contents, image vector feature extraction is performed on the information to be retrieved to obtain features to be retrieved; for example, feature extraction is performed using methods such as a vision transformer (ViT) model. The present disclosure selects different feature extraction methods according to different similarity detection tasks to be applicable to various image similarity query tasks.

[0064] It is easy for those skilled in the art to understand that, for other types of similarity query tasks, the present disclosure may also use a matching feature extraction method to extract features, and the present disclosure does not impose any special limitation on the feature extraction method for the features to be retrieved.

[0065] In an exemplary embodiment of the present disclosure, a clustering feature library can be constructed by the following steps: obtaining a pre-constructed image database, the image database including multiple images; based on pre-configured feature categories, performing feature extraction processing on the multiple images to construct multiple feature databases, the feature categories including one or more of image pixel features, image semantic features and image content features; performing feature clustering processing on the multiple images in each feature database to obtain a clustering feature library.

[0066] The image database, also known as the retrieval image library, may be a database for storing images. The image database is only used to store images and does not classify or otherwise process the images. The feature category may be a feature category used for clustering images. For example, the feature category may include but is not limited to pixel features, image content features, image semantic features, etc.

[0067] refer to Figure 4 , Figure 4 The overall flow chart of performing feature matching based on a clustering feature library to determine similar images according to an exemplary embodiment of the present disclosure is schematically shown. A pre-built image database is obtained, and feature extraction can be performed on multiple images in the image database. When extracting features, it can be performed according to pre-configured feature categories, such as Figure 4 In the method, pixel features, image content features and image semantic features are extracted for each image respectively to obtain the image features corresponding to each image.

[0068] For different feature categories corresponding to multiple images, corresponding feature databases can be constructed. For example, for multiple images that have been processed by pixel feature extraction, corresponding feature database 1 is generated; for multiple images that have been processed by image content feature extraction, corresponding feature database 2 is generated; for multiple images that have been processed by image semantic feature extraction, corresponding feature database 3 is generated.

[0069] After extracting the image features corresponding to each of the multiple images, the multiple images in each feature database are subjected to feature clustering processing, and the feature database after clustering processing is used as a cluster feature library. By extracting features of different feature categories from the image, a cluster feature library can be constructed so that the cluster feature library can be used for feature matching to reduce the amount of image retrieval calculation.

[0070] In other exemplary embodiments of the present disclosure, more feature databases of different feature categories may be constructed, and the present disclosure does not impose any special limitation on the feature categories and the number of feature databases.

[0071] In an exemplary embodiment of the present disclosure, feature clustering processing is performed on multiple images in each feature database to obtain a cluster feature library, including: determining the feature distance between different images in the feature database; performing feature clustering processing on the multiple images according to the feature distance to obtain multiple image categories with different image features, and the feature distance between any two images in the image category is less than a distance threshold.

[0072] The image category may be a cluster formed by one or more images whose image feature similarity is within a certain specified similarity interval. The feature distance may be the distance between the image feature vectors of two images.

[0073] By using pre-configured feature categories, feature extraction processing is performed on multiple images in the image database to obtain multiple feature databases. For the images in the feature database, feature clustering processing can be further performed, such as by calculating the feature distance between any two images in the feature database. The image feature vector corresponding to each image in the feature database is obtained, and feature clustering is performed on the images in each feature database based on the image feature vector, such as using cosine similarity (cos distance) or Euclidean distance to calculate the feature distance.

[0074] When the feature distance between two images is less than the distance threshold, that is, the image similarity is less than the specified threshold, the two images can be considered to belong to the same category. Through the above method, multiple images in the feature database are naturally clustered into several categories. Finally, the feature distance between any two images in each image category is less than the distance threshold.

[0075] refer to Figure 5 , Figure 5 The flowchart of building a cluster feature library according to an exemplary embodiment of the present disclosure is schematically shown. In the process of building a cluster feature library, for the image categories in the cluster feature library, the first image feature under each feature category can be taken as the category standard feature of the category. For a new image feature obtained, the image feature can be matched one by one with all the category standard features in a feature database.

[0076] If the image feature matches a standard category feature, the image and the image feature are added to a similar category group; if the image feature does not match all the category standard features in the feature database, a new category group can be established based on the image feature, and the image feature is used as the standard category feature of the category group. Repeat the above steps until multiple cluster feature libraries with different features are constructed. The cluster feature library is constructed by feature distance, so that the images in each image category in the cluster feature library have a high similarity, which is convenient for performing subsequent image retrieval tasks.

[0077] In an exemplary embodiment of the present disclosure, for step S230, based on multiple category image features, a similar image category matching the feature to be retrieved is determined, including: determining a first feature distance between the feature to be retrieved and each category image feature; based on the first feature distance, selecting similar image features from the category image features; and taking the image category in which the similar image features are located as the similar image category.

[0078] The category image feature may be a category standard feature corresponding to each image category. The first feature distance may be a distance between the feature to be retrieved and each category image feature.

[0079] Continue to refer Figure 3 After extracting the features to be retrieved, in step S330, the category features are matched in the cluster feature library. The features to be retrieved are compared with the category image features corresponding to each image category contained in the cluster feature library, and the feature distance between the two is calculated as the first feature distance, and then similar image features similar to the first feature distance are screened from multiple category image features.

[0080] For example, (1) for the image or text that needs to be queried in the information to be retrieved, feature extraction processing is performed to extract the features that need to be queried, that is, the features to be retrieved. The features to be retrieved may include pixel features, semantic features, recognition content features, etc., and the features are recorded as fquery.

[0081] (2) For the pre-built cluster feature library, the category image features f1, f2, f3...FN, etc. of all N image categories in the corresponding cluster feature library can be retrieved. (3) The feature distances between the query fquery and all N category features are calculated respectively, and other feature distances such as Euclidean distance or cosine distance are calculated as the first feature distance.

[0082] (4) From the calculated multiple first feature distances, select the n categories (generally n=3, because the larger the n, the greater the amount of calculation) of category features with the shortest distance to fquery as similar image features. After determining the similar image features through the above steps, similar images can be retrieved only from the similar image features, which greatly reduces the amount of calculation for feature matching.

[0083] In an exemplary embodiment of the present disclosure, for step S240, similar images corresponding to the information to be retrieved are determined according to the similar image categories, including: obtaining candidate image features corresponding to each image under each similar image category; determining a second feature distance between the feature to be retrieved and the candidate image feature; and generating a similar image list based on the distance value of the second feature distance, wherein the similar image list includes one or more similar images.

[0084] The candidate image feature may be a feature corresponding to each image included in the similar image category. The second feature distance may be a distance between the feature to be retrieved and each candidate image feature under the similar image category. The similar image list may be a display list generated by one or more similar images.

[0085] Continue to refer Figure 3 In step S340, the feature to be retrieved is matched with the image in the N categories with the highest similarity. After determining multiple similar image categories, the image features corresponding to each image contained in each similar image category are obtained as candidate image features. The second feature distance between the feature to be retrieved and the candidate image features is calculated respectively. For example, the Euclidean distance (or other feature distance) is calculated between the feature to be retrieved fquery and all the image features in n categories (such as n=2, 3, ...) of similar image categories to obtain the second feature distance.

[0086] According to the calculated second feature distance, a similarity list is outputted from small to large, and then a similar image list is generated, and in step S350, a similar image sequence is outputted. Since in the image retrieval process, only the similarity between the feature to be retrieved and the most similar similar image categories is calculated, the feature distance calculation is reduced, and the image retrieval calculation amount is reduced.

[0087] It should be noted that the terms “first”, “second”, etc. used in the present disclosure are only used to distinguish different characteristic distances and should not impose any limitations on the present disclosure.

[0088] In summary, the similar image determination method disclosed in the present invention obtains information to be retrieved, performs feature extraction processing on the information to be retrieved, and obtains the features to be retrieved of the information to be retrieved; obtains a pre-configured cluster feature library, and the cluster feature library includes category image features corresponding to multiple image categories; determines the similar image category matching the features to be retrieved according to the multiple category image features; and determines the similar images corresponding to the information to be retrieved according to the similar image category. On the one hand, for the features to be retrieved, first compare them with the image features in the cluster feature library to determine the similar image category, and then determine the similar images from the similar image category, which can greatly reduce the amount of query calculation and improve the image query efficiency. On the other hand, by determining similar images from similar image categories, it is only necessary to compare with several categories of category features with relatively similar features, which can reduce the feature distance calculation and the amount of image retrieval calculation. The larger the image library, the more time is saved. On the other hand, the present invention can be applied to different similarity query tasks and perform accurate queries in a variety of different similarity query tasks.

[0089] It should be noted that, although the steps of the method of the present invention are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0090] In addition, in this exemplary embodiment, a similar image determination device is also provided. Figure 6 The similar image determination device 600 may include: a retrieval information acquisition module 610 , a feature library acquisition module 620 , a similar category determination module 630 and a similar image determination module 640 .

[0091] Specifically, the retrieval information acquisition module 610 is used to obtain the information to be retrieved, perform feature extraction processing on the information to be retrieved, and obtain the features to be retrieved of the information to be retrieved; the feature library acquisition module 620 is used to obtain a pre-configured cluster feature library, and the cluster feature library includes category image features corresponding to multiple image categories; the similar category determination module 630 is used to determine the similar image category that matches the features to be retrieved based on multiple category image features; the similar image determination module 640 is used to determine the similar images corresponding to the information to be retrieved based on the similar image category.

[0092] In an exemplary embodiment of the present disclosure, the retrieval information acquisition module 610 includes a retrieval feature extraction unit for: determining image retrieval task information corresponding to the information to be retrieved; and performing feature extraction processing on the information to be retrieved based on the image retrieval task information to obtain features to be retrieved.

[0093] In an exemplary embodiment of the present disclosure, the retrieval feature extraction unit includes a retrieval feature extraction sub-unit, which is used to: when the image retrieval task information is to retrieve images with the same semantics, perform image feature extraction and text feature extraction on the information to be retrieved to obtain the features to be retrieved; when the image retrieval task information is to retrieve images with similar pixels, perform structural feature extraction and texture feature extraction on the information to be retrieved to obtain the features to be retrieved; when the image retrieval task information is to retrieve images with similar contents, perform image vector feature extraction on the information to be retrieved to obtain the features to be retrieved.

[0094] In an exemplary embodiment of the present disclosure, the feature library acquisition module 620 also includes a feature library construction unit, which is used to: obtain a pre-constructed image database, the image database includes multiple images; based on pre-configured feature categories, perform feature extraction processing on the multiple images to construct multiple feature databases, the feature categories include one or more of image pixel features, image semantic features and image content features; perform feature clustering processing on the multiple images in each feature database to obtain a clustered feature library.

[0095] In an exemplary embodiment of the present disclosure, a clustering feature library includes multiple image categories, and a feature library construction unit includes a feature library construction sub-unit, which is used to: determine the feature distance between different images in the feature database; perform feature clustering processing on the multiple images according to the feature distance to obtain multiple image categories with different image features, and the feature distance between any two images in the image category is less than a distance threshold.

[0096] In an exemplary embodiment of the present disclosure, the similarity category determination module 630 includes a feature determination unit, which is used to: determine a first feature distance between a feature to be retrieved and each category image feature; select similar image features from the category image features based on the first feature distance; and use the image category in which the matching similar image features are located as the similar image category.

[0097] In an exemplary embodiment of the present disclosure, the similar image determination module 640 includes a similar image determination unit, which is used to: obtain candidate image features corresponding to each image under each similar image category; determine the second feature distance between the feature to be retrieved and the candidate image feature; and generate a similar image list based on the distance value of the second feature distance, wherein the similar image list contains one or more similar images.

[0098] The specific details of the virtual modules of each similar image determination device mentioned above have been described in detail in the corresponding similar image determination method. The undisclosed details can be found in the implementation content of the method part, so they will not be repeated here.

[0099] It should be noted that although several modules or units of the similar image determination device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0100] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0101] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0102] The exemplary embodiments of the present disclosure further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the similar image determination method is implemented.

[0103] In one embodiment, the computer program product may be a tangible product containing the computer program, such as a computer-readable storage medium storing the computer program. Figure 7 , Figure 7 The schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure is schematically shown. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, and other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid state drive (SSD), etc. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory (Nand Flash), etc.

[0104] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0105] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0106] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure, such as the similar image determination method described above.

[0107] The exemplary embodiments of the present disclosure also provide an electronic device, which may include a processor and a memory. The memory stores executable instructions of the processor, such as a computer program. The processor executes the method steps of various exemplary embodiments of the present disclosure by executing the executable instructions. In addition, the electronic device may also include a display for displaying a graphical user interface.

[0108] Reference below Figure 8 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Figure 8 The electronic device 800 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0109] like Figure 8 As shown, the electronic device 800 may include: a processor 810 , a memory 820 , a bus 830 , an I / O (input / output) interface 840 , a network adapter 850 , and a display 860 .

[0110] The memory 820 may include a volatile memory, such as a RAM 821, a cache unit 822, and may also include a non-volatile memory, such as a ROM 823. The memory 820 may also include one or more program modules 824, such program modules 824 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 824 may include each module in the above-mentioned device.

[0111] The processor 810 may include one or more processing units. For example, the processor 810 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0112] The processor 810 may be used to execute executable instructions stored in the memory 820 , such as executing the above-mentioned similar image determination method.

[0113] The bus 830 is used to realize the connection between different components of the electronic device 800, and may include a data bus, an address bus, and a control bus.

[0114] The electronic device 800 can communicate with one or more external devices 900 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 840 .

[0115] The electronic device 800 can communicate with one or more networks through the network adapter 850. For example, the network adapter 850 can provide mobile communication solutions such as 3G / 4G / 5G, or wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 850 can communicate with other modules of the electronic device 800 through the bus 830.

[0116] The electronic device 800 may display a graphical user interface, such as a display interface showing similar images, through the display 860 .

[0117] although Figure 8 Not shown, other hardware and / or software modules may also be provided in the electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0118] As can be seen from the above, the technical solution of the present disclosure can be implemented as a method, an apparatus, a system, a computer program product, a storage medium, an electronic device, etc. Those skilled in the art can understand that various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, such as being respectively referred to as a "circuit", "module" or "system".

[0119] It should be understood that the present disclosure is not limited to the specific method steps or structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. Those skilled in the art will easily think of other embodiments based on the specific embodiments provided by the present disclosure. Therefore, the specific embodiments provided by the present disclosure are only exemplary, and the scope and spirit of the present disclosure are indicated by the claims, and any variations, uses or adaptive changes of the present disclosure should be covered, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the technical field that are not disclosed in the present disclosure.

Claims

1. A similar image determination method, characterized in that: include: Acquire information to be retrieved, perform feature extraction processing on the information to be retrieved, and obtain features to be retrieved of the information to be retrieved; Acquire a preconfigured clustering feature library, wherein the clustering feature library includes category image features corresponding to each of a plurality of image categories; Determine, based on the plurality of the category image features, a similar image category matching the feature to be retrieved; According to the similar image category, similar images corresponding to the information to be retrieved are determined.

2. The method according to claim 1, characterized in that: The step of performing feature extraction processing on the information to be retrieved to obtain features to be retrieved of the information to be retrieved includes: Determine the image retrieval task information corresponding to the information to be retrieved; Based on the image retrieval task information, feature extraction processing is performed on the information to be retrieved to obtain the features to be retrieved.

3. The method according to claim 2, characterized in that The step of performing feature extraction processing on the information to be retrieved based on the image retrieval task information to obtain the features to be retrieved includes: When the image retrieval task information is to retrieve images with the same semantics, image feature extraction and text feature extraction are performed on the information to be retrieved to obtain the features to be retrieved; When the image retrieval task information is to retrieve images with similar pixels, structural feature extraction and texture feature extraction are performed on the information to be retrieved to obtain the features to be retrieved; When the image retrieval task information is to retrieve images with similar contents, image vector features are extracted from the information to be retrieved to obtain the features to be retrieved.

4. The method according to claim 1, characterized in that: The step of obtaining a pre-configured clustering feature library includes: Acquire a pre-built image database, wherein the image database includes a plurality of images; Based on pre-configured feature categories, feature extraction processing is performed on the multiple images to construct multiple feature databases, where the feature categories include one or more of image pixel features, image semantic features, and image content features; Perform feature clustering processing on each of the multiple images in the feature database to obtain a cluster feature library.

5. The method according to claim 4, characterized in that The cluster feature library includes multiple image categories, and the feature clustering process is performed on multiple images in each of the feature databases to obtain a cluster feature library, including: Determining feature distances between different images in the feature database; The plurality of images are subjected to feature clustering processing according to the feature distance to obtain a plurality of image categories with different image features, wherein the feature distance between any two images in the image category is less than a distance threshold.

6. The method according to claim 1, characterized in that The step of determining the similar image category corresponding to the feature to be retrieved based on the plurality of category image features comprises: Determine a first feature distance between the feature to be retrieved and each of the category image features; Selecting similar image features from the category image features according to the first feature distance; The image category in which the similar image features are located is used as the similar image category.

7. The method according to claim 1 or 6, characterized in that: The determining, according to the similar image category, similar images corresponding to the information to be retrieved includes: Obtaining candidate image features corresponding to each image in each of the similar image categories; Determining a second feature distance between the feature to be retrieved and the candidate image feature; A similar image list is generated based on the distance value of the second characteristic distance, wherein the similar image list includes one or more similar images.

8. A similar image determination device, characterized in that: include: A search information acquisition module is used to acquire the information to be searched, perform feature extraction processing on the information to be searched, and obtain the features to be searched of the information to be searched; A feature library acquisition module is used to acquire a pre-configured cluster feature library, wherein the cluster feature library includes category image features corresponding to each of a plurality of image categories; A similar category determination module, used to determine a similar image category matching the feature to be retrieved based on the plurality of category image features; The similar image determination module is used to determine the similar images corresponding to the information to be retrieved according to the similar image categories.

9. An electronic device, characterized in that: include: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the method for determining similar images as claimed in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the similar image determination method according to any one of claims 1 to 7 is implemented.

Images