Image sorting method and device, computer readable storage medium and terminal equipment

Abstract

The invention belongs to the technical field of image retrieval, and particularly relates to an image sorting method and device, a computer readable storage medium and terminal equipment. The method comprises the steps of obtaining a first retrieval sorting result of a to-be-queried image; wherein the first retrieval sorting result comprises each first image and a first similarity between each first image and the to-be-queried image; calculating each second similarity corresponding to each first image; wherein the second similarity of the first image is the similarity of the first image and a second image, and the first similarity of the second image is greater than the first similarity of the first image; and based on the first similarity and the second similarity of each first image, determining a second retrieval sorting result of the to-be-queried image.

Description

Technical Field

[0001] The present application belongs to the field of image retrieval technology, and in particular, relates to an image sorting method, apparatus, computer-readable storage medium, and terminal device. Background Art

[0002] Image retrieval is an important research direction in the field of computer vision and multimedia. It involves retrieving images that are similar or related to the query image from a large-scale image database. In the usual image retrieval process, features are extracted from the query image and each image in the image database to obtain the corresponding image features. Then, the similarity between the image features of the query image and the image features of each image in the image database can be calculated; then, the calculated similarities are sorted to determine the retrieval result of the query image. However, the image sorting method in the existing image retrieval process only roughly considers the features of the query image, resulting in low accuracy of image retrieval. Summary of the invention

[0003] In view of this, the embodiments of the present application provide an image sorting method, apparatus, computer-readable storage medium and terminal device to solve the problem that the image sorting method in the existing image retrieval process is relatively rough, resulting in low accuracy of image retrieval.

[0004] A first aspect of an embodiment of the present application provides an image sorting method, which may include:

[0005] Obtaining a first search ranking result of the image to be queried; wherein the first search ranking result includes each first image and a first similarity between each first image and the image to be queried;

[0006] Calculating respective second similarities corresponding to respective first images; wherein the second similarity of the first image is the similarity between the first image and the second image, and the first similarity of the second image is greater than the first similarity of the first image;

[0007] Based on the first similarities and the second similarities of the first images, a second retrieval ranking result of the image to be queried is determined.

[0008] A second aspect of the embodiments of the present application provides an image sorting device, which may include:

[0009] An acquisition module, configured to acquire a first search ranking result of the image to be queried; wherein the first search ranking result includes each first image and a first similarity between each first image and the image to be queried;

[0010] A calculation module, used to calculate respective second similarities corresponding to respective first images; wherein the second similarity of the first image is the similarity between the first image and the second image, and the first similarity of the second image is greater than the first similarity of the first image;

[0011] The determination module is used to determine a second retrieval ranking result of the to-be-queried image based on the first similarity and the second similarity of each of the first images.

[0012] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned image sorting methods are implemented.

[0013] A fourth aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned image sorting methods when executing the computer program.

[0014] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of any one of the above-mentioned image sorting methods.

[0015] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain the first retrieval ranking result of the image to be queried; wherein the first retrieval ranking result includes each first image and each first image and the first similarity of the image to be queried; calculate each second similarity corresponding to each first image; wherein the second similarity of the first image is the similarity of the first image and the second image, and the first similarity of the second image is greater than the first similarity of the first image; based on the first similarity and the second similarity of each first image, determine the second retrieval ranking result of the image to be queried. Through the embodiments of the present application, on the basis of the initial retrieval ranking result (first retrieval ranking result), the similarity of each (first) image in the initial ranking result and the (second) image ranked before the image can be further calculated, which helps to identify images that are ranked low in the initial retrieval ranking result but have high similarity with multiple positive samples (i.e., images belonging to the same category as the image to be queried), and reduce the influence of negative samples ranked high, thereby improving the recall rate, which can greatly improve the accuracy of image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 Schematic diagram of the image retrieval process;

[0018] Figure 2 This is a flow chart of an embodiment of an image sorting method in an embodiment of the present application;

[0019] Figure 3 is a schematic diagram of the first similarity and the second similarity;

[0020] Figure 4 is a schematic diagram of the first similarity and the third similarity;

[0021] Figure 5 This is a structural diagram of an embodiment of an image sorting device in an embodiment of the present application;

[0022] Figure 6 This is a schematic block diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0025] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0026] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0027] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0028] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0029] Image retrieval is an important research direction in the field of computer vision and multimedia. It involves retrieving images that are similar or related to the query image from a large-scale image database. In the usual image retrieval process, features are extracted from the query image and each image in the image database to obtain the corresponding image features, and then the similarity between the image features of the query image and the image features of each image in the image database is calculated; then, the calculated similarities are sorted, and the retrieval result of the query image can be determined based on the sorting result.

[0030] For example, see Figure 1 , feature extraction can be performed on the query image to obtain the query feature; feature extraction can also be performed on each image in the image database to obtain feature 1 corresponding to image 1, feature 2 corresponding to image 2, ..., feature n corresponding to image n; then, the similarity between the query feature and feature 1, feature 2, ..., feature n can be calculated respectively; then, the calculated similarities can be sorted in descending order to obtain the sorting result of the similarity of each image in the image database.

[0031] However, the existing image ranking methods in the image retrieval process only roughly consider the characteristics of the query image, resulting in low accuracy of image retrieval.

[0032] In view of this, the embodiments of the present application provide an image sorting method, apparatus, computer-readable storage medium and terminal device to solve the problem that the image sorting method in the existing image retrieval process is relatively rough, resulting in low accuracy of image retrieval.

[0033] It should be noted that the execution subject of the method of the present application is a terminal device, specifically, it can be a common computing device such as a desktop computer, a notebook, a PDA, a server, a mobile phone, etc., or it can be other computing devices.

[0034] In an embodiment of the present application, based on the initial sorting results, the similarities between the images in the sorting results can be further considered, so that the global information of each image in the sorting results can be fully utilized, and a more accurate sorting result can be obtained, laying the foundation for high-precision image retrieval.

[0035] The following is a detailed description of the image sorting in the embodiment of the present application. Figure 2 , an embodiment of an image sorting method in the embodiment of the present application may include:

[0036] Step S201: Obtain a first search ranking result of the image to be queried.

[0037] In the embodiment of the present application, the image that needs to be retrieved may be referred to as a query image. Specifically, an image that is the same as or similar to the query image may be searched in a preset image database; here, the image that is the same as or similar to the query image may be an image of the same category as the query image. The image database may include various pre-collected and stored images.

[0038] In an embodiment of the present application, feature extraction may be first performed on the query image to obtain image features of the query image (referred to as query features); feature extraction may also be performed on each image in the image database to obtain image features corresponding to each image.

[0039] Here, a preset image feature extraction model can be used to extract features of the query image and each image in the image database; the image feature extraction model can be any artificial intelligence model for image feature extraction in the prior art, and the image feature extraction model can include but is not limited to any artificial intelligence model such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Autoencoders, Deep Residual Networks (ResNet), YOLO series, Transformer network, etc., and the embodiments of the present application are not limited to this.

[0040] In a specific implementation, since each time an image search is performed in an image database, it is necessary to extract features from each image in the image database, the features corresponding to each image may also be stored in the image database to improve the efficiency of image search.

[0041] After the feature to be queried and the features of each image in the image database are obtained, the similarities between the feature to be queried and the image features of each image in the image database may be calculated to obtain each similarity.

[0042] Afterwards, the images in the image database can be sorted based on the similarities to obtain an initial sorting result; if the image feature of an image in the image database has a greater similarity with the feature to be queried, the image will be ranked higher in the initial sorting result; if the image feature of an image in the image database has a smaller similarity with the feature to be queried, the image will be ranked lower in the initial sorting result.

[0043] The similarity between the above image features can preferably be cosine similarity, or any index that characterizes the degree of difference between images; for example, the similarity between the above image features can be an index such as the Euclidean distance or Manhattan distance between the image features. In addition, unless otherwise specified, the similarity between two images hereinafter refers to the similarity between the image features of the two images.

[0044] Based on the initial ranking results of each image in the image database, a first search ranking result of the image to be queried can be determined. The first search ranking result may include each (specifically N) image (referred to as the first image) and the similarity between each image and the image to be queried (referred to as the first similarity).

[0045] In an embodiment of the present application, the first image may only include some images in the image database; specifically, the top N images with the greatest similarity to the image to be queried in the initial sorting results may be used as each first image, and each first image only includes N images in the image database; wherein, the value of N can be set according to actual needs, and the embodiment of the present application is not limited to this, for example, N can be set to 7, 10, 20, etc.

[0046] Step S202: Calculate the second similarities corresponding to the first images respectively.

[0047] In real scenes, images of the same category may be diverse. For example, different face images may contain faces from different angles, although they are all face images. Another example is that different body images may contain bodies from different views (front, back or side). Another example is that images of certain still objects (such as plants and animals) may show diversity due to different shooting angles and distances. In addition, the image feature extraction model itself may also have limitations, which may cause some positive samples (images belonging to the same category as the query image) to be ranked at the bottom in the first retrieval ranking result, while some negative samples (images belonging to different categories from the query image) are ranked at the top, thereby affecting the accuracy of image retrieval.

[0048] Therefore, in the embodiment of the present application, the first search ranking result can also be optimized and adjusted based on the relationship between the first images to obtain a second search ranking result with higher accuracy.

[0049] Specifically, the second similarities corresponding to each first image can be calculated; taking any first image as an example (referred to as the target first image), other first images (referred to as second images) that are ranked before the target first image can be determined from the first retrieval ranking result; that is, each first image in the first retrieval ranking result whose first similarity is greater than the target first similarity (the first similarity of the target first image) can be used as the second image.

[0050] For example, the first search ranking result is image 4, image 2, image 5, image 7, image 1, ..., image N. If the target first image is image 7, and the images before image 7 in the first search ranking result are image 4, image 2, and image 5, then image 4, image 2, and image 5 can be used as the second images corresponding to image 7; if the target first image is image 1, and the images before image 1 in the first search ranking result are image 4, image 2, image 5, and image 7, then image 4, image 2, image 5, and image 7 can be used as the second images corresponding to image 1.

[0051] Afterwards, the similarity between the target first image and each corresponding second image (referred to as second similarity) may be calculated.

[0052] For each first image, the corresponding second similarities can be calculated according to the above method, thereby obtaining the relationship (ie, the second similarities) between each first image and each first image that is ranked before the first image in the first search ranking result.

[0053] Step S203: Determine a second search ranking result of the image to be queried based on the first similarity and the second similarity of each first image.

[0054] Afterwards, the first images can be re-sorted based on the first similarities and second similarities corresponding to the first images to obtain a second search ranking result. Here, for each first image, the third similarity corresponding to the first image can be determined based on the first similarity and second similarities corresponding to the first image, and then the first images can be sorted based on the third similarities corresponding to the first images to obtain a second search ranking result.

[0055] As an example, the first search ranking result may include 7 first images, and in descending order of the corresponding first similarities, the first images are image 1, image 2, image 3, image 4, image 5, image 6 and image 7; in this example, please refer to Figure 3 In the 8*7 grid shown, each row and each column in the grid can correspond to an image; here, the first row can correspond to the image to be queried, the second row and the first column can correspond to image 1, the third row and the second column can correspond to image 2, the fourth row and the third column can correspond to image 3, the fifth row and the fourth column can correspond to image 4, the sixth row and the fifth column can correspond to image 5, the seventh row and the sixth column can correspond to image 6, and the eighth row and the seventh column can correspond to image 7; each grid unit in the grid can represent the similarity between the image corresponding to the row of the grid unit and the image corresponding to the column; for example, the grid unit corresponding to the first row and the first column can represent the similarity between the image corresponding to the first row (i.e., the image to be queried) and the image corresponding to the first column (i.e., image 1). Therefore, the first row of the grid can represent each first similarity.

[0056] In the above example, for each first image, each second similarity of the first image can be calculated; here, the second image corresponding to image 2 is image 1, then the second similarity between image 2 and image 1 can be calculated (i.e. Figure 3 The second image corresponding to image 3 is image 1 and image 2, then the second similarity between image 3 and image 1 can be calculated (i.e. Figure 3 The grid cell corresponding to the second row and third column in the image), the second similarity between image 3 and image 2 (i.e. Figure 3 The grid unit corresponding to the third row and the third column in ); and so on, the second similarities corresponding to each first image can be calculated.

[0057] In the above example, for each first image, the third similarity of the first image can be determined based on the first similarity of the first image and each second similarity; for example, the third similarity of the first image can be determined based on the first similarity of image 2 (i.e. Figure 3 The grid cell corresponding to the first row and second column in the graph) and the second similarity (i.e. Figure 3The third similarity corresponding to image 2 can be determined based on the grid cell corresponding to the second row and the second column in the image 2. For another example, the first similarity corresponding to image 3 (i.e. Figure 3 The grid cell corresponding to the first row and third column in the Figure 3 The grid unit corresponding to the second row and the third column and the grid unit corresponding to the third row and the third column in the image are used to determine the third similarity corresponding to image 3.

[0058] In the embodiment of the present application, specifically, the first weight and each second weight may be used to perform weighted averaging on the first similarities and the second similarities corresponding to each first image to obtain the fourth similarities corresponding to each first image.

[0059] Among them, the first weight is the weight corresponding to the first similarity, and the second weight is the weight corresponding to the corresponding second similarity; the specific values ​​of the first weight and the second weight can be concretized and situationally set according to actual needs, and the embodiments of the present application are not limited to this.

[0060] In the embodiment of the present application, the fourth similarity S1 of the i-th first image in the first search ranking result i The calculation formula can be shown as follows:

[0061]

[0062] Among them, c 1i is the first weight corresponding to the first similarity of the i-th first image, S0 i is the first similarity of the i-th first image, M ij is the similarity between the ith first image and the jth first image (here, the second similarity of the ith first image), c 2j is the corresponding second similarity (i.e., M ij ) corresponds to the second weight.

[0063] In a specific implementation, the value of the first weight can be set to the first similarity of the corresponding first image, and the value of the second weight can be set to the first similarity of the corresponding second image; for example, when calculating the fourth similarity of the i-th first image, the first similarity of the i-th first image and the second similarities of the i-th first image can be weighted averaged; at this time, the first weight c 1i Set the first similarity S0 of the i-th first image i , the second weight c 2j Set the first similarity (S0) of the corresponding second image (i.e., the jth first image) j ); In this setting, the calculation formula of the fourth similarity of the i-th first image can be expressed as:

[0064]

[0065] In another specific implementation, the values ​​of the first weight and the second weight may both be set to 1, and the calculation formula for the fourth similarity of the i-th first image may be expressed as:

[0066]

[0067] After the fourth similarity is determined, a second search ranking result of the image to be queried may be determined based on each first image and the corresponding fourth similarity.

[0068] Specifically, the first images may be sorted in descending order according to the corresponding fourth similarities, and the sorting result is the second search sorting result.

[0069] By calculating the fourth similarity, the mutual relationship of each first image can be integrated to obtain more comprehensive information, thereby better resisting the interference of negative samples and obtaining more accurate retrieval ranking results.

[0070] In a specific implementation method of an embodiment of the present application, after obtaining the second retrieval ranking result, the second retrieval ranking result can be directly used as the retrieval ranking result finally returned; based on the second retrieval ranking result, the image retrieval result of the image to be queried can be determined; for example, the top M first images in the second retrieval ranking result can be used as the image retrieval results of the image to be queried, and the value of M can be set according to actual needs.

[0071] In another specific implementation method of the embodiment of the present application, K representative first images (called third images) can also be anchored from each first image; wherein, the value of K can be set according to actual needs, and the embodiment of the present application is not limited to this, for example, K can be set to 2 or 3, etc.; thereafter, the similarity between each first image and the anchored K images (called the third similarity) can be calculated, and based on the first similarities corresponding to each first image and each third similarity, the third retrieval ranking result of the image to be queried can be determined.

[0072] The above implementation method will be introduced in detail below.

[0073] Specifically, the first K first images with the largest first similarities in the first search ranking result can be determined as the third images. Since the first similarities of the third images are large, it can be considered that the determined third images are likely to be positive samples, so that the third images can be used as representatives of positive samples, and the third similarities between the first images and the third images can be calculated.

[0074] As an example, the first search ranking result may include 7 first images, and in descending order of the corresponding first similarities, the first images are image 1, image 2, image 3, image 4, image 5, image 6 and image 7; in this example, you can refer to Figure 4 The 8*7 grid shown in the figure, each row and each column in the grid can correspond to an image, and the specific correspondence is as follows Figure 3 The grid is the same as that of , and will not be repeated here; the first row of the grid can represent each first similarity.

[0075] In the above example, for each first image, each third similarity of the first image can be calculated; here, K can be set to 2, that is, the first two first images (i.e., image 1 and image 2) with the largest first similarity in the first search ranking result can be determined as the third image; when calculating each third similarity corresponding to image 1, the third similarity between image 1 and image 1 (i.e., Figure 4 The grid cell corresponding to the second row and first column in ), the third similarity between image 1 and image 2 (i.e. Figure 4 When calculating the third similarities corresponding to image 2, the third similarity between image 2 and image 1 can be calculated (i.e. Figure 4 The grid cell corresponding to the second row and second column in ), the third similarity between image 2 and image 2 (i.e. Figure 4 When calculating the third similarities corresponding to image 3, the third similarity between image 3 and image 1 can be calculated (i.e. Figure 4 The grid cell corresponding to the second row and third column in ), the third similarity between image 3 and image 2 (i.e. Figure 4 The grid unit corresponding to the third row and the third column in ); and so on, the third similarities corresponding to each first image can be calculated.

[0076] In the above example, for each first image, the fifth similarity of the first image can be determined based on the first similarity of the first image and each third similarity; for example, the fifth similarity of the first image can be determined based on the first similarity of image 1 (i.e. Figure 4 The grid cell corresponding to the first row and first column in the Figure 4 The fifth similarity corresponding to image 1 can be determined based on the grid cell corresponding to the first column of the second row and the grid cell corresponding to the first column of the third row). For another example, the fifth similarity corresponding to image 2 can be determined based on the first similarity (i.e. Figure 4 The grid cell corresponding to the first row and second column in the Figure 4 The grid cell corresponding to the second row and the second column, and the grid cell corresponding to the third row and the second column, determine the fifth similarity corresponding to image 2.

[0077] In the embodiment of the present application, specifically, the third weight and the fourth weight may be used to perform weighted averaging on the first similarities and the third similarities corresponding to the first images to obtain the fifth similarities corresponding to the first images.

[0078] Among them, the third weight is the weight corresponding to the first similarity, and the fourth weight is the weight corresponding to the corresponding third similarity; the specific values ​​of the third weight and the fourth weight can be concretized and situationally set according to actual needs, and the embodiments of the present application are not limited to this.

[0079] In the embodiment of the present application, the fifth similarity S2 of the i-th first image in the first search ranking result i The calculation formula can be shown as follows:

[0080]

[0081] Among them, c 3i is the third weight corresponding to the first similarity of the i-th first image, M ij is the similarity between the i-th first image and the j-th first image (here, the third similarity of the i-th first image), c 4j M ij The corresponding fourth weight.

[0082] In a specific implementation, the value of the third weight can be set to the first similarity of the corresponding first image, and the value of the fourth weight can be set to the first similarity of the corresponding third image; for example, when calculating the fifth similarity of the i-th first image, the first similarity of the i-th first image and the third similarities of the i-th first image can be weighted averaged; at this time, the third weight c 3i Set the first similarity S0 of the i-th first image i , the fourth weight c 4j Set the first similarity (S0) of the corresponding third image (ie, the jth first image) j ); In this setting, the calculation formula of the fifth similarity of the i-th first image can be expressed as:

[0083]

[0084] In another specific implementation, the values ​​of the third weight and the fourth weight can be set to 1, and the calculation formula of the fifth similarity of the i-th first image can be expressed as:

[0085]

[0086] After the fifth similarity is determined, a third search ranking result of the image to be queried may be determined based on each first image and the corresponding fifth similarity.

[0087] Specifically, the first images may be sorted in descending order according to the corresponding fifth similarities, and the sorting result is the third search sorting result.

[0088] By calculating the fifth similarity, the similarity between each first image and the positive sample can be strengthened, thereby improving the ranking of the positive sample and reducing the ranking of the negative sample, which helps to reduce the deviation caused by feature extraction errors and obtain more robust retrieval ranking results.

[0089] In a specific implementation method of an embodiment of the present application, after obtaining the third retrieval ranking result, the third retrieval ranking result can be directly used as the final returned retrieval ranking result; based on the third retrieval ranking result, the image retrieval result of the image to be queried can be determined; for example, the top M first images in the third retrieval ranking result can be used as the image retrieval result of the image to be queried.

[0090] In another specific implementation of the embodiment of the present application, the fourth search ranking result of the image to be queried can also be determined based on the first similarity, the second similarity, and the third similarity corresponding to each first image. Accordingly, the mutual relationship between the first images and the similarity between the first image and the positive sample image can be comprehensively considered to more comprehensively evaluate the similarity between the first image and the image to be queried, which can enhance the accuracy and robustness of image retrieval at the same time.

[0091] The above specific implementation method will be introduced in detail below.

[0092] Specifically, the sixth similarities corresponding to the first images may be determined based on the first similarities, the second similarities, and the third similarities corresponding to the first images.

[0093] Here, the fifth weight and the sixth weight can be used to perform weighted averaging on the fourth similarities and fifth similarities corresponding to each first image to obtain the sixth similarities corresponding to each first image; wherein the fifth weight is the weight corresponding to the fourth similarity, and the sixth weight is the weight corresponding to the fifth similarity.

[0094] The specific values ​​of the fifth weight and the sixth weight can be concretized and set according to actual needs, and the embodiments of the present application are not limited to this.

[0095] In the embodiment of the present application, the sixth similarity S3 of the i-th first image in the first search ranking result i The calculation formula can be shown as follows:

[0096] S3 i =(c 5i *S1 i +c 6i *S2 i ) / (c 5i +c 6i ),i=1,2,...,N,

[0097] Among them, c 5i is the fifth weight corresponding to the fourth similarity of the i-th first image, c 6i is the sixth weight corresponding to the fifth similarity of the i-th first image.

[0098] In a specific implementation, the fifth weight and the sixth weight can be set to 0.5, then the sixth similarity S3 of the i-th first image is i The calculation formula can be expressed as:

[0099] S3 i =(S1 i +S2 i ) / 2,i=1,2,...,N.

[0100] After the sixth similarity is determined, a fourth search ranking result of the image to be queried may be determined based on each first image and the corresponding sixth similarity.

[0101] Specifically, the first images may be sorted in descending order according to the corresponding sixth similarities, and the sorting result is the fourth search sorting result.

[0102] In a specific implementation method of an embodiment of the present application, after obtaining the fourth retrieval ranking result, the fourth retrieval ranking result can be directly used as the final returned retrieval ranking result; based on the fourth retrieval ranking result, the image retrieval result of the image to be queried can be determined; for example, the top M first images in the fourth retrieval ranking result can be used as the image retrieval result of the image to be queried.

[0103] Based on this, the advantages of the above two implementation methods can be fully combined to obtain more accurate and robust image retrieval results.

[0104] It should be understood that in actual image retrieval applications, any one of the above retrieval ranking results can be flexibly selected as the final returned retrieval ranking result as needed, so as to better adapt to actual application scenarios.

[0105] Since the image sorting method in the embodiment of the present application is an unsupervised algorithm and does not involve model training, it can be applied to existing image retrieval systems as a general post-processing method in image retrieval scenarios, thereby significantly improving the usability and scene adaptability of the image retrieval method.

[0106] In summary, the embodiment of the present application obtains the first retrieval ranking result of the image to be queried; wherein, the first retrieval ranking result includes each first image and each first image's first similarity with the image to be queried; calculates each second similarity corresponding to each first image; wherein, the second similarity of the first image is the similarity between the first image and the second image, and the first similarity of the second image is greater than the first similarity of the first image; based on the first similarity and the second similarity of each first image, determines the second retrieval ranking result of the image to be queried. Through the embodiment of the present application, on the basis of the initial retrieval ranking result (first retrieval ranking result), the similarity of each (first) image in the initial ranking result with the (second) image ranked before the image can be further calculated, which helps to identify images that are ranked low in the initial retrieval ranking result but have high similarity with multiple positive samples (i.e., images belonging to the same category as the image to be queried), and reduce the influence of negative samples ranked high, thereby improving the recall rate, which can greatly improve the accuracy of image retrieval.

[0107] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0108] Corresponding to an image sorting method described in the above embodiment, Figure 5 A structural diagram of an embodiment of an image sorting device provided in an embodiment of the present application is shown.

[0109] In an embodiment of the present application, an image sorting device may include:

[0110] An acquisition module 501 is used to acquire a first search ranking result of the image to be queried; wherein the first search ranking result includes each first image and a first similarity between each first image and the image to be queried;

[0111] A calculation module 502 is used to calculate respective second similarities corresponding to respective first images; wherein the second similarity of the first image is the similarity between the first image and the second image, and the first similarity of the second image is greater than the first similarity of the first image;

[0112] The determination module 503 is configured to determine a second search ranking result of the to-be-queried image based on the first similarity and the second similarity of each of the first images.

[0113] In a specific implementation of the embodiment of the present application, the device further includes:

[0114] A second calculation module is used to calculate the third similarity between each of the first images and each of the third images; wherein each of the third images is the first K first images with the largest first similarity in the first search ranking result;

[0115] The second determination module is used to determine a third retrieval ranking result of the image to be queried based on the first similarities and the third similarities respectively corresponding to each of the first images.

[0116] In a specific implementation of the embodiment of the present application, the device further includes:

[0117] The third determination module is used to determine a fourth retrieval ranking result of the image to be queried based on the first similarity, the second similarity and the third similarity respectively corresponding to each of the first images.

[0118] In a specific implementation of the embodiment of the present application, the determining module includes:

[0119] A first weighted average submodule, configured to perform weighted averaging of the first similarities and the second similarities corresponding to each of the first images using a first weight and a second weight, to obtain a fourth similarity corresponding to each of the first images; wherein the first weight is a weight corresponding to the first similarity, and the second weight is a weight corresponding to the second similarity;

[0120] The first determination submodule is used to determine the second search ranking result of the image to be queried based on each of the first images and the corresponding fourth similarities.

[0121] In a specific implementation of the embodiment of the present application, the second determining module includes:

[0122] A second weighted average submodule is used to perform weighted averaging on the first similarities and the third similarities corresponding to each of the first images using a third weight and a fourth weight, so as to obtain a fifth similarity corresponding to each of the first images; wherein the third weight is a weight corresponding to the first similarity, and the fourth weight is a weight corresponding to the third similarity;

[0123] The second determining submodule is used to determine the third retrieval ranking result of the image to be queried based on each of the first images and the corresponding fifth similarities.

[0124] In a specific implementation of the embodiment of the present application, the third determining module includes:

[0125] a third weighted average submodule, configured to use a fifth weight and a sixth weight to perform weighted averaging on the fourth similarities and the fifth similarities corresponding to each of the first images, to obtain the sixth similarities corresponding to each of the first images; wherein the fifth weight is the weight corresponding to the fourth similarity, and the sixth weight is the weight corresponding to the fifth similarity;

[0126] The third determining submodule is used to determine the fourth search ranking result of the image to be queried based on each of the first images and the corresponding sixth similarities.

[0127] In a specific implementation of the embodiment of the present application, the device further includes:

[0128] A third determining module, configured to determine an image retrieval result of the image to be queried based on the second retrieval ranking result of the image to be queried;

[0129] a fourth determining module, configured to determine the image retrieval result of the image to be queried based on the third retrieval ranking result of the image to be queried;

[0130] The fifth determining module is used to determine the image retrieval result of the image to be queried based on the fourth retrieval ranking result of the image to be queried.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0132] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0133] Figure 6 A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0134] like Figure 6 As shown, the terminal device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned various image sorting method embodiments are implemented, for example Figure 2Alternatively, when the processor 60 executes the computer program 62, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of modules 501 to 503 are shown.

[0135] Exemplarily, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 62 in the terminal device 6.

[0136] The terminal device 6 may be a computing device such as a desktop computer, a notebook, a PDA, a smart phone, or a smart TV. Those skilled in the art will appreciate that Figure 6 It is only an example of the terminal device 6 and does not constitute a limitation of the terminal device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 6 may also include input and output devices, network access devices, buses, etc.

[0137] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 60 may be the nerve center and command center of the terminal device 6. The processor 60 may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0138] The memory 61 may be an internal storage unit of the terminal device 6, such as a hard disk or memory of the terminal device 6. The memory 61 may also be an external storage device of the terminal device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 6. Further, the memory 61 may also include both an internal storage unit of the terminal device 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the terminal device 6. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0139] The terminal device 6 may also include a communication module, which may provide communication solutions including wireless local area networks (WLAN) (such as Wi-Fi networks), Bluetooth, Zigbee, mobile communication networks, global navigation satellite systems (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to network devices. The communication module may be one or more devices integrating at least one communication processing module. The communication module may include an antenna, which may have only one array element or an antenna array including multiple array elements. The communication module may receive electromagnetic waves through the antenna, frequency modulate and filter the electromagnetic wave signals, and send the processed signals to the processor. The communication module may also receive the signal to be sent from the processor, frequency modulate and amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0140] The terminal device 6 may further include a power management module, which may receive input from an external power source, a battery and / or a charger to power the processor, the memory, the communication module, and the like.

[0141] The terminal device 6 may further include a display module, which may be used to display information input by a user or information provided to a user. The display module may include a display panel, and optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel may cover the display panel, and when the touch panel detects a touch operation on or near it, it is transmitted to the processor to determine the type of the touch event, and then the processor provides a corresponding visual output on the display panel according to the type of the touch event.

[0142] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0148] An embodiment of the present application provides a computer program product. When the computer program product is executed on the terminal device, the terminal device can implement the steps in the above-mentioned method embodiments.

[0149] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0150] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image sorting method, characterized in that: include: Obtaining a first search ranking result of the image to be queried; wherein the first search ranking result includes each first image and a first similarity between each first image and the image to be queried; Calculating respective second similarities corresponding to respective first images; wherein the second similarity of the first image is the similarity between the first image and the second image, and the first similarity of the second image is greater than the first similarity of the first image; Based on the first similarities and the second similarities of the first images, a second retrieval ranking result of the image to be queried is determined.

2. The image sorting method according to claim 1, characterized in that: Also includes: Calculating the third similarity between each of the first images and each of the third images; wherein each of the third images is the first K first images with the largest first similarity in the first search ranking result; A third retrieval ranking result of the image to be queried is determined based on the first similarities and the third similarities respectively corresponding to the first images.

3. The image sorting method according to claim 2, characterized in that: Also includes: A fourth retrieval ranking result of the image to be queried is determined based on the first similarities, the second similarities, and the third similarities respectively corresponding to the first images.

4. The image sorting method according to claim 3, characterized in that: The determining, based on the first similarity and the second similarity of each of the first images, a second retrieval ranking result of the image to be queried comprises: Using the first weight and the second weight, weighted average is performed on the first similarities and the second similarities corresponding to each of the first images, so as to obtain the fourth similarities corresponding to each of the first images; wherein the first weight is the weight corresponding to the first similarity, and the second weight is the weight corresponding to the second similarity; Based on each of the first images and the corresponding fourth similarities, the second retrieval ranking result of the image to be queried is determined.

5. The image sorting method according to claim 4, characterized in that: The determining, based on the first similarities and the third similarities respectively corresponding to the first images, a third search ranking result of the image to be queried includes: Using the third weight and the fourth weight, weighted average is performed on the first similarities and the third similarities corresponding to each of the first images, so as to obtain the fifth similarities corresponding to each of the first images; wherein the third weight is the weight corresponding to the first similarity, and the fourth weight is the weight corresponding to the third similarity; The third retrieval ranking result of the image to be queried is determined based on each of the first images and the corresponding fifth similarities.

6. The image sorting method according to claim 5, characterized in that: The determining of the fourth search ranking result of the image to be queried based on the first similarities, the second similarities, and the third similarities respectively corresponding to the first images includes: Using the fifth weight and the sixth weight, weighted average is performed on the fourth similarities and the fifth similarities corresponding to each of the first images, so as to obtain the sixth similarities corresponding to each of the first images; wherein the fifth weight is the weight corresponding to the fourth similarity, and the sixth weight is the weight corresponding to the fifth similarity; The fourth search ranking result of the image to be queried is determined based on each of the first images and the corresponding sixth similarities.

7. The image sorting method according to claim 3, characterized in that: Also includes: Determining an image retrieval result of the image to be queried based on the second retrieval ranking result of the image to be queried; Alternatively, based on the third search ranking result of the image to be queried, determining the image search result of the image to be queried; Alternatively, the image retrieval result of the image to be queried is determined based on the fourth retrieval ranking result of the image to be queried.

8. An image sorting device, characterized in that: include: An acquisition module, configured to acquire a first search ranking result of the image to be queried; wherein the first search ranking result includes each first image and a first similarity between each first image and the image to be queried; A calculation module, used to calculate respective second similarities corresponding to respective first images; wherein the second similarity of the first image is the similarity between the first image and the second image, and the first similarity of the second image is greater than the first similarity of the first image; The determination module is used to determine a second retrieval ranking result of the to-be-queried image based on the first similarity and the second similarity of each of the first images.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the image sorting method according to any one of claims 1 to 7 are implemented.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image sorting method according to any one of claims 1 to 7 are implemented.

Images