Image processing, retrieval methods, apparatus, devices and storage media

Abstract

This application provides an image processing and retrieval method, apparatus, device, and storage medium to improve the efficiency of image retrieval. The method includes: forming at least one image pair using a source image and similar images from a set of similar images, wherein the set of similar images includes at least one similar image; performing feature matching on the source image and similar images in the image pair to determine corresponding first feature subsets and second feature subsets; determining the similarity between the first feature subsets and the second feature subsets based on the topological information of feature points; and determining a similar image filtering result from the set of similar images based on the similarity. This method has low computational complexity, low resource consumption, and can improve the retrieval efficiency of similar images.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to an image processing method and apparatus, an image retrieval method and apparatus, a method for searching for commodity objects, a road search method, an electronic device, and a storage medium. Background Technology

[0002] Content-Based Image Retrieval (CBIR) is a method for finding images based on their content.

[0003] In one example, a user can input an image, and based on CBIR technology, other images with the same or similar content can be found. Current CBIR technology typically involves identifying feature points in two images, estimating a transformation matrix based on these feature points, matching the changes in the feature points against the transformation matrix, determining a set of feature points, and then repeating the transformation matrix process until convergence or an error requirement is met, thus identifying similar images that meet the criteria.

[0004] However, the above method requires multiple iterative calculations to obtain similar images, which is time-consuming and resource-intensive. Summary of the Invention

[0005] This application provides an image processing method to improve the efficiency of image retrieval.

[0006] Accordingly, embodiments of this application also provide an image processing apparatus, an image retrieval method and apparatus, a product object search method, a road search method, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0007] To address the aforementioned problems, this application discloses an image processing method, comprising: forming at least one image pair using a source image and similar images from a set of similar images of the source image, wherein the set of similar images includes at least one similar image; performing feature matching on the source image and similar images in the image pair to determine a corresponding first feature subset and a second feature subset; determining the similarity between the first feature subset and the second feature subset based on the topological structure information of feature points; and determining a similar image filtering result in the set of similar images based on the similarity.

[0008] This application also discloses an image retrieval method, the method comprising: receiving a retrieval request, the retrieval request including a source image; performing a retrieval in a graph database based on the content of the source image to determine a set of similar images of the source image, the set of similar images including at least one similar image; using the source image and the similar images in the set of similar images to form at least one image pair; performing feature matching on the source image and the similar images in the image pair to determine a corresponding first feature subset and a second feature subset; determining the similarity between the first feature subset and the second feature subset based on the topological structure information of feature points; filtering the similar images in the set of similar images according to the similarity; determining a retrieval result based on the filtered similar images; and sending the retrieval result.

[0009] This application also discloses a method for searching for product objects. The method includes: receiving a product object search request, the product object search request including an image of a product object; performing a search based on the image of the product object to determine a set of similar images of the image of the product object, the set of similar images including at least one similar image of a similar product object; using the image of the product object and the similar images in the set of similar images to form at least one image pair; performing feature matching on the image of the product object and the similar images in the image pair to determine a corresponding first feature subset and a second feature subset; determining the similarity between the first feature subset and the second feature subset based on the topological structure information of feature points; filtering the similar images in the set of similar images according to the similarity; determining product object information of the corresponding similar product object based on the filtered similar images; and sending the product object information of the similar product object.

[0010] This application also discloses a road search method, the method comprising: receiving a road search request, the road search request including a road image; performing a retrieval based on the road image to determine a set of similar images of the road image, the set of similar images including at least one similar product object; using the road image and the similar images in the set of similar images to form at least one image pair; performing feature matching on the road image and the similar images in the image pair to determine a corresponding first feature subset and a second feature subset; determining the similarity between the first feature subset and the second feature subset based on the topological structure information of feature points; filtering the similar images in the set of similar images according to the similarity; determining the road information corresponding to the filtered similar images; and sending the road information.

[0011] This application also discloses an image processing apparatus, comprising: an image pair determination module, configured to form at least one image pair using a source image and similar images from a set of similar images of the source image, the set of similar images including at least one similar image; a feature matching module, configured to perform feature matching on the source image and similar images in the image pair to determine corresponding first feature subsets and second feature subsets; a similarity determination module, configured to determine the similarity between the first feature subset and the second feature subset based on the topological structure information of feature points; and a sorting module, configured to determine the similar image filtering result in the set of similar images based on the similarity.

[0012] This application also discloses an image retrieval device, comprising: a request receiving module for receiving a retrieval request, the retrieval request including a source image; a coarse ranking module for performing a retrieval in a graph database based on the content of the source image to determine a set of similar images of the source image, the set of similar images including at least one similar image; a fine ranking module for forming at least one image pair using the source image and similar images in the set of similar images; performing feature matching on the source image and similar images in the image pair to determine a corresponding first feature subset and a second feature subset; determining the similarity between the first feature subset and the second feature subset based on the topological structure information of feature points; filtering similar images in the set of similar images according to the similarity; and a result returning module for determining a retrieval result based on the filtered similar images and sending the retrieval result.

[0013] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform one or more methods as described in this application.

[0014] This application also discloses one or more machine-readable media storing executable code thereon, which, when executed, causes a processor to perform one or more methods as described in this application.

[0015] Compared with the prior art, the embodiments of this application have the following advantages:

[0016] In this embodiment, at least one image pair is formed by using a source image and similar images from a set of similar images of the source image. Feature matching is performed on the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset. Based on the topological structure information of the feature points, the similarity between the first feature subset and the second feature subset is determined. The similarity of the topological structure is used to verify the similar images. This method has low computational load and low resource consumption, thereby improving the retrieval efficiency of similar images when determining the similar image filtering results in the set of similar images based on the similarity. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating an example of image processing according to an embodiment of this application;

[0018] Figure 2 This is a flowchart illustrating the steps of an embodiment of an image processing method according to this application;

[0019] Figure 3 This is a flowchart illustrating the steps of an embodiment of an image retrieval method according to this application;

[0020] Figure 4 This is a flowchart illustrating the steps of an embodiment of a product object search method according to this application;

[0021] Figure 5 This is a flowchart illustrating the steps of an embodiment of the road search method of this application;

[0022] Figure 6 This is a structural block diagram of an embodiment of an image processing apparatus according to this application;

[0023] Figure 7 This is a structural block diagram of an optional embodiment of an image retrieval device according to this application;

[0024] Figure 8 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0025] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] The embodiments of this application can be applied to various fields of image content retrieval, such as computer vision and graph databases. For example, it can be used for road image retrieval to quickly locate a position, or for product image retrieval to quickly retrieve product objects.

[0027] Reference Figure 1 A schematic diagram illustrating an example of image processing is shown.

[0028] Step 102: Use the source image and similar images from a set of similar images of the source image to form at least one image pair, wherein the set of similar images includes at least one similar image.

[0029] Content-based image retrieval can be divided into two stages: coarse ranking and fine ranking. In the coarse ranking stage, potentially similar images are quickly found from large datasets such as databases, forming a set of similar images. In the fine ranking stage, similar images in the set retrieved in the coarse ranking stage are further matched with the source images to filter out truly similar images and remove images falsely recalled in the coarse ranking stage.

[0030] Therefore, in one optional embodiment, a source image is obtained; based on the content of the source image, a search is performed to determine a set of similar images of the source image. The source image to be searched can be obtained through a search request, etc., and then, in the coarse-sorting stage, at least one similar image of the source image is retrieved from the corresponding database to form a set of similar images. Then, the fine-sorting stage is performed based on this set of similar images.

[0031] In this approach, each similar image in the source image and the set of similar images can be paired together. For example, if the set of similar images contains n similar images, then n image pairs can be formed. If the source image is denoted as q and the similar images as d, then the coarse-ranked list obtained by coarsely ranking the source image can be: [q, (d1, d2, ..., d...]. n )], where (d1,d2,……,d n (q,d1), (q,d2), ..., (q,d3) represents a set of similar images. Correspondingly, similar images and source images can be extracted to form image pairs: [(q,d1), (q,d2), ..., (q,d4)]. n )).

[0032] like Figure 1 In the example, each similar image is extracted sequentially from the set of similar images, and then this similar image and the source image are combined to form an image pair. Thus, a set of similar images with n similar images can form n image pairs. Figure 1 In the example, n>5.

[0033] Step 104: Perform feature matching on the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset.

[0034] For each image pair (q,d) i ), where 1≤i≤n, can be used to perform geometric verification based on the similarity of the topological structures of two images to complete the fine sorting.

[0035] This process involves comparing the source and similar images in an image pair using feature extraction. This feature comparison calculates similarity by extracting feature points and calculating their topological structure. Specifically, feature point sets for each image are determined through feature extraction. Then, feature points from the two sets are matched, and the matched feature points are used to define the corresponding feature subsets.

[0036] In one optional embodiment, the step of performing feature matching on the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset includes: extracting features from the source image and similar images in the image pair to determine the corresponding first feature set and second feature set; and performing feature point matching based on the first feature set and second feature set to determine the first feature subset and second feature subset.

[0037] Feature extraction can be performed on the source image and similar images separately. In one example, local feature extraction can be performed based on the Speeded-Up Robust Features (SURF) technique to obtain the corresponding feature points and their feature information, including descriptors, coordinates, etc., where the descriptors are used to describe the feature points. In other examples, local feature extraction can also be performed based on the Scale-Invariant Feature Transform (SIFT) algorithm, or on deep local feature algorithms, etc., depending on the specific requirements. This application does not impose any limitations on this approach.

[0038] Local feature extraction can be performed on the source image to obtain multiple corresponding feature points and their feature information. The extracted feature points can be used to form a first feature set, denoted as S. q Local feature extraction is performed on similar images to obtain multiple feature points and their feature information. These extracted feature points can be used to construct a second feature set, denoted as S. d For each image pair, feature extraction can yield a first feature set of the source image and a second feature set of similar images.

[0039] Then, feature points from the first feature set (referred to as first feature points) and feature points from the second feature set (referred to as second feature points) are used for matching. The matching of feature points can be performed based on the descriptors of the corresponding feature points. For a certain first feature point P in the first feature set... qi A second feature point P in the second feature set dj Based on the first feature point P qi The descriptor and a feature point P in the second feature set djFeature point matching is performed on the descriptor, and whether a feature point matches can be determined based on corresponding matching rules. For example, the distance between two feature points can be determined based on their positions in the image, thus determining whether they meet the matching rules. In one example, the matching rule for two feature points is that they are each other's nearest neighbors; in another example, the matching rule for two feature points is that they are each other's nearest neighbors, and the nearest neighbor distance / second nearest neighbor distance < a distance threshold, such as a distance threshold of 0.7, etc. The distance threshold can be set based on experience. If the matching rule is met, the two feature points match, and these two feature points can be placed into their respective feature subsets of the image. Here, the first feature subset is a subset of the first feature set, and the second feature subset is a subset of the second feature set. If a first feature point and a second feature point meet the matching rule, the first feature point can be placed into the first feature subset S. q In ', the second feature point is placed into the second feature subset S'. d ', where S q 'For S q A subset of S d 'For S d A subset of.

[0040] like Figure 1 The feature points within the circular region defined by the midpoint line constitute the feature subsets corresponding to image pair i, including the first feature subset corresponding to the source image and the second feature subset corresponding to the similar image.

[0041] Step 106: Determine the similarity between the first feature subset and the second feature subset based on the topological structure information of the feature points.

[0042] After determining the feature subsets (first feature subset and second feature subset) of the feature points that match the source image and its similar images, geometric verification can be performed based on the similarity of the topological structure of the feature points. Therefore, the topological structure information of the feature points corresponding to the first feature subset and the topological structure information of the feature points corresponding to the second feature subset can be determined, and then the similarity between the first feature subset and the second feature subset can be determined.

[0043] In one optional embodiment, determining the similarity between the first feature subset and the second feature subset based on the topological information of feature points includes: determining the topological information of feature points in the first feature subset to obtain a corresponding first vector; determining the topological information of feature points in the second feature subset to obtain a corresponding second vector; and determining the similarity between the first vector and the second vector. The topological structure formed by the feature points in the feature subset can be determined based on the coordinates of each feature point, and then the topological structures in the two feature subsets can be compared to determine similarity. Specifically, the topological structure formed by the feature points can be represented by vectors, thus using vectors as topological information to calculate similarity.

[0044] The step of determining the topological structure information of feature points in the feature subset to obtain the corresponding vector includes: determining the Euclidean distance between feature points in the feature subset, and determining the corresponding distance vector based on the Euclidean distance; normalizing the distance vector to obtain the corresponding vector, which is used to characterize the topological structure of the feature subset. The feature subset includes a first feature subset and a second feature subset; the feature points include a first feature point and a second feature point. For each feature subset, the Euclidean distance between each pair of feature points based on coordinates can be calculated, and then the Euclidean distance can be used as the dimension value of the corresponding dimension in the vector to obtain the corresponding distance vector. After normalization to eliminate the influence of image scale on the vector, the corresponding vector is obtained, which is used to characterize the topological structure of the feature subset.

[0045] In a further optional embodiment, determining the Euclidean distance between feature points in the feature subset and determining the corresponding distance vector based on the Euclidean distance includes: sorting the feature points in the feature subset according to a set order; determining the Euclidean distance between each pair of feature points; and sorting the Euclidean distances according to the set order to obtain the corresponding distance vector.

[0046] The feature points in the feature subset can be sorted according to a set order, so that the sorting order of the corresponding matching feature points in different feature subsets is the same. Then, for the feature points in the feature subset, the Euclidean distance between each pair of feature points is calculated. Then, according to the sorting order of the feature points, the Euclidean distance is used as the dimension value of the vector in that dimension. Assuming there are m feature points in the feature subset, the dimension of the vector is (m-1)*m / 2.

[0047] Based on the above embodiments, determining the topological structure information of feature points in the first feature subset to obtain the corresponding first vector includes: determining the first Euclidean distance between feature points in the first feature subset, and determining the corresponding first distance vector based on the first Euclidean distance; normalizing the first distance vector to obtain the corresponding first vector, wherein the first vector is used to characterize the topological structure of the first feature subset. The step of determining the first Euclidean distance between feature points in the first feature subset and determining the corresponding first distance vector based on the first Euclidean distance includes: sorting the feature points in the first feature subset according to a set order; determining the first Euclidean distance between each pair of feature points; and sorting the first Euclidean distance according to the set order to obtain the corresponding first distance vector.

[0048] Determining the topological structure information of feature points in the second feature subset to obtain the corresponding second vector includes: determining the second Euclidean distance between feature points in the second feature subset, and determining the corresponding second distance vector based on the second Euclidean distance; normalizing the second distance vector to obtain the corresponding second vector, which is used to characterize the topological structure of the second feature subset. The step of determining the second Euclidean distance between feature points in the second feature subset and determining the corresponding second distance vector based on the second Euclidean distance includes: sorting the feature points in the second feature subset according to a set order; determining the second Euclidean distance between each pair of feature points; and sorting the second Euclidean distance according to the set order to obtain the corresponding second distance vector.

[0049] As in the example above, for the first feature subset S q For all feature points in the array (assuming there are m points in total), a predetermined order (S) can be determined first. d (The order in which feature points are set during calculation is the same as this order, ensuring that the sorting order of matching feature points is the same.) Then, the first Euclidean distance r between the coordinates of each pair of feature points is calculated. ij This can be understood as the distance between the i-th feature point and the j-th feature point, which is then arranged in order to obtain the first distance vector v1. For example, the first m-1 dimensions of v are r. 12 ,r 13 ,…,r 1m Then m-2 dimensions are r 23 ,r 24 ,…,r 2m Similarly, the dimension of the first distance vector v1 is (m-1)*m / 2. Normalizing the first distance vector v1 yields the representation S. q The first vector v of the topological structure q For the second feature subset S d 'Corresponding to the second vector v dThe calculation is similar to that described above.

[0050] After determining the first vector and the second vector, the similarity between the first vector and the second vector can be calculated. In one optional embodiment, determining the similarity between the first vector and the second vector includes determining the cosine similarity between the first vector and the second vector. The similarity between the first vector and the second vector can be determined by determining the cosine value of the angle between the first vector and the second vector.

[0051] In this embodiment, Euclidean distance and cosine similarity are used as examples to illustrate the similarity between two feature subsets. In actual processing, other similarity methods can also be used, such as Manhattan distance, correlation distance, etc. This embodiment does not limit this.

[0052] Step 108: Based on the similarity, determine the similar image filtering results in the similar image set.

[0053] For each image pair, the similarity between the included source image and similar images can be calculated. Therefore, the similar images of the source image can be sorted according to this similarity score to obtain the corresponding similar image filtering results. Furthermore, based on these similar image filtering results, similar images can be filtered further; for example, the top x images can be selected as target images, and the corresponding search results can be returned, x... <n。

[0054] In one optional embodiment, determining the similar image filtering result in the similar image set based on the similarity includes: determining the similarity score between the source image and the similar images based on the similarity; sorting the similar images in the similar image set according to their corresponding similarity scores to obtain the similar image filtering result. In the fine-ranking stage, the similarity of image pairs can be used as the similarity score of the corresponding similar images. This similarity score is used to determine the degree of similarity between the two images and can also be used as the score in the fine-ranking stage to determine the fine-ranking result. In other examples, similar images may also correspond to descriptive information such as labels, thereby determining weights based on the descriptive information, and determining similarity scores based on the weights and similarity. Thus, each similar image in the similar image set corresponds to a similarity score, and then the similar images in the similar image set can be sorted according to the size of the similarity score. For example, the larger the similarity score, the higher the similarity, and the higher the ranking of the similar image, thus obtaining the similar image filtering result for the source image in the fine-ranking stage. In some optional embodiments, the similar images can also be filtered based on the similarity score, for example, filtering similar images with similarity scores greater than a similarity threshold into the similar image filtering result, and providing feedback on the filtered similar images.

[0055] For example, in image retrieval scenarios, the top x similar images with the highest similarity scores can be selected as target images, generating corresponding search results and providing feedback. These results can include the target image and its descriptive information, such as name and title, or a link to the target image, depending on the specific requirements. In other examples, similar images with similarity scores greater than or equal to a specific similarity threshold can be selected.

[0056] In summary, by using a source image and similar images from a set of similar images to form at least one image pair, feature matching is performed on the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset. Based on the topological structure information of the feature points, the similarity between the first feature subset and the second feature subset is determined. Similarity is verified based on the similarity of the topological structure. This method has low computational load and low resource consumption, thereby enabling the determination of similar image filtering results in the set of similar images based on the similarity, thus improving the retrieval efficiency of similar images.

[0057] Based on the above embodiments, this application also provides an image processing method that selects a set of similar images in the coarse ranking stage, and calculates the topological structure based on feature points in the image in the fine ranking stage, thereby completing the fine ranking by geometric verification through the similarity of the topological structure.

[0058] Reference Figure 2 The diagram shows a flowchart of the steps of an embodiment of an image processing method according to this application.

[0059] Step 202: Obtain the source image.

[0060] Step 204: Based on the content of the source image, perform a search to determine a set of similar images of the source image, wherein the set of similar images includes at least one similar image.

[0061] Step 206: Use the source image and similar images from the set of similar images of the source image to form at least one image pair.

[0062] For each image pair, the following steps 208-216 can be performed:

[0063] Step 208: Perform feature extraction on the source image to determine the corresponding first feature set, and perform feature extraction on similar images to determine the corresponding second feature set.

[0064] Step 210: Perform feature point matching based on the first feature set and the second feature set to determine the matched feature point pairs.

[0065] Step 212: Extract feature points from the first feature set in the feature point pair to form a first feature subset, and extract feature points from the second feature set in the feature point pair to form a second feature subset.

[0066] Step 214: Determine the topological structure information of the feature points in the first feature subset to obtain the corresponding first vector; and determine the topological structure information of the feature points in the second feature subset to obtain the corresponding second vector.

[0067] The process of determining the topological structure information of feature points in the first feature subset and obtaining the corresponding first vector includes: determining the first Euclidean distance between feature points in the first feature subset, and determining the corresponding first distance vector based on the first Euclidean distance; normalizing the first distance vector to obtain the corresponding first vector, wherein the first vector is used to characterize the topological structure of the first feature subset. The step of determining the first Euclidean distance between feature points in the first feature subset and determining the corresponding first distance vector based on the first Euclidean distance includes: sorting the feature points in the first feature subset according to a predetermined order; determining the first Euclidean distance between each pair of feature points; and sorting the first Euclidean distance according to the predetermined order to obtain the corresponding first distance vector.

[0068] Determining the topological structure information of feature points in the second feature subset to obtain the corresponding second vector includes: determining the second Euclidean distance between feature points in the second feature subset, and determining the corresponding second distance vector based on the second Euclidean distance; normalizing the second distance vector to obtain the corresponding second vector, which is used to characterize the topological structure of the second feature subset. The step of determining the second Euclidean distance between feature points in the second feature subset and determining the corresponding second distance vector based on the second Euclidean distance includes: sorting the feature points in the second feature subset according to a set order; determining the second Euclidean distance between each pair of feature points; and sorting the second Euclidean distance according to the set order to obtain the corresponding second distance vector.

[0069] Step 216: Determine the similarity between the first vector and the second vector. This allows the similarity between the source image and similar images to be determined for each image pair.

[0070] Step 218: Determine the similarity score between the source image and the similar image based on the similarity.

[0071] Step 220: Sort the similar images in the similar image set according to their corresponding similarity scores to obtain the similar image filtering results.

[0072] Step 222: Filter the similar images based on the similarity score and return the filtered similar images.

[0073] The background technique estimates the transformation matrix of feature points, then selects a set of point pairs S, and iteratively executes this process. This requires calculating the transformation points of all points in A under the current transformation matrix, resulting in a large computational load. In contrast, the embodiments of this application, based on a first and second feature subset, only require calculating the distance between each pair of feature points to obtain the topological structure representation vector, significantly reducing the computational load. Furthermore, the background technique requires sampling, which can lead to unstable results and discrepancies between multiple calculations. The embodiments of this application, utilizing vector similarity calculations, ensure consistent results across multiple calculations, resulting in higher accuracy.

[0074] Furthermore, the time complexity of the method described in the background technique is O(tk+tn), where t is the number of iterations, k is the average number of points involved in the transformation matrix calculation in each iteration, and n is the number of input feature point pairs. For image matching, t >> n is usually the case, and due to the matrix inversion operation involved, the constant term of the time complexity is also very large. In contrast, the time complexity of the method in the embodiment of this application is only O(n^2), which is more efficient.

[0075] The embodiments described above can be applied to various execution entities, such as filtering similar images on a client-side device or on a server-side device such as a server, cloud device, or edge computing device. The specific application can be determined based on the application scenario and requirements. For example, in some scenarios, the filtering can be performed on either the client-side or server-side device. In other scenarios, a coarse sorting can be performed on the client-side device, and then the image identifiers corresponding to the coarsely sorted image set can be transmitted to the server-side device. The server-side device can then perform a fine sorting on the coarse sorting results to obtain the similar image filtering results.

[0076] Based on the above embodiments, this application also provides an image retrieval method that can provide content-based image retrieval.

[0077] Reference Figure 3 The diagram shows a flowchart of an embodiment of an image retrieval method according to this application.

[0078] Step 302: Receive a retrieval request, the retrieval request including the source image.

[0079] Step 304: Based on the content of the source image, a search is performed in the graph database to determine a set of similar images of the source image, wherein the set of similar images includes at least one similar image.

[0080] Step 306: Use the source image and similar images from the set of similar images to form at least one image pair.

[0081] Step 308: Perform feature matching on the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset.

[0082] Step 310: Based on the topological structure information of the feature points, determine the similarity between the first feature subset and the second feature subset.

[0083] Step 312: Based on the similarity, filter the similar images in the set of similar images.

[0084] Step 314: Determine the search results based on the similar images obtained from the screening, and send the search results.

[0085] The steps in this embodiment are similar to those in the above embodiments, and you can refer to the description of the above embodiments for details.

[0086] Therefore, in content-based image retrieval, geometric verification based on topological similarity can be used to complete fine ranking, thereby improving the processing efficiency and accuracy of image retrieval.

[0087] Based on the above embodiments, this application also provides a method for searching product objects, which can provide image-based product object search functions on e-commerce websites, apps, etc.

[0088] Reference Figure 4 The diagram illustrates a flowchart of an embodiment of a product object search method according to this application.

[0089] Step 402: Receive a product object search request, wherein the product object search request includes an image of a product object.

[0090] Step 404: Based on the image of the product object, perform a search to determine a set of similar images of the product object, wherein the set of similar images includes at least one similar image of a similar product object.

[0091] Step 406: Use the image of the product object and similar images from the set of similar images to form at least one image pair.

[0092] Step 408: Perform feature matching on the image of the product object and similar images in the image pair to determine the corresponding first feature subset and second feature subset.

[0093] Step 410: Based on the topological structure information of the feature points, determine the similarity between the first feature subset and the second feature subset.

[0094] Step 412: Based on the similarity, filter the similar images in the set of similar images.

[0095] Step 414: Based on the similar images obtained from the screening, determine the product object information of the corresponding similar product objects.

[0096] Step 416: Send the product object information of the similar product object.

[0097] The steps in this embodiment are similar to those in the above embodiments, and you can refer to the description of the above embodiments for details.

[0098] In e-commerce websites and apps, users can take pictures of similar or identical products they want to buy and quickly find them through image search. Based on the images of products in the database of products on the e-commerce website, similar or identical products can be quickly retrieved and feedback can be provided, improving the user's search efficiency and meeting user needs.

[0099] Based on the above embodiments, this application also provides a road search method that can search roads based on road images, such as road surveillance images, to quickly retrieve accident locations, missing persons locations, etc.

[0100] Reference Figure 5 The diagram shows a flowchart of an embodiment of a road search method according to this application.

[0101] Step 502: Receive a road search request, which includes road images.

[0102] Step 504: Based on the road image, perform a search to determine a set of similar images of the road image, wherein the set of similar images includes at least one similar image of a similar product object.

[0103] Step 506: Use the road image and similar images from the set of similar images to form at least one image pair.

[0104] Step 508: Perform feature matching on the road image and similar images in the image pair to determine the corresponding first feature subset and second feature subset.

[0105] Step 510: Determine the similarity between the first feature subset and the second feature subset based on the topological structure information of the feature points.

[0106] Step 512: Based on the similarity, filter the similar images in the set of similar images.

[0107] Step 514: Determine the road information corresponding to the similar images obtained from the filtering.

[0108] Step 516: Send the road information, such as the road name and the corresponding latitude and longitude coordinates.

[0109] The steps in this embodiment are similar to those in the above embodiments, and you can refer to the description of the above embodiments for details.

[0110] In the event of an accident, or when a distress image is received on a social media platform, forum, or in an emergency call, the system can provide image content-based retrieval based on road images, such as those from road surveillance cameras, to quickly locate the corresponding road in the image. This facilitates rapid resolution of related incidents and improves safety.

[0111] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0112] Based on the above embodiments, this embodiment also provides an image processing device that can be applied to electronic devices such as terminal devices and servers.

[0113] Reference Figure 6 The diagram illustrates a structural block diagram of an embodiment of an image processing apparatus according to this application, which may specifically include the following modules:

[0114] The image pair determination module 602 is used to form at least one image pair by using a source image and similar images from a set of similar images of the source image, wherein the set of similar images includes at least one similar image.

[0115] The feature matching module 604 is used to perform feature matching between the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset.

[0116] The similarity determination module 606 is used to determine the similarity between the first feature subset and the second feature subset based on the topological structure information of the feature points.

[0117] The sorting module 608 is used to determine the similar image filtering results in the similar image set based on the similarity.

[0118] The feature matching module 604 is used to extract features from the source image and similar images in the image pair to determine the corresponding first feature set and second feature set; and to perform feature point matching based on the first feature set and second feature set to determine the first feature subset and second feature subset.

[0119] The similarity determination module 606 includes: a vector determination submodule and a determination submodule, wherein:

[0120] The vector determination submodule is used to determine the topological structure information of feature points in the first feature subset to obtain the corresponding first vector; and to determine the topological structure information of feature points in the second feature subset to obtain the corresponding second vector.

[0121] The determining submodule is used to determine the similarity between the first vector and the second vector.

[0122] The vector determination submodule is used to determine the Euclidean distance between feature points in the feature subset, and determine the corresponding distance vector based on the Euclidean distance; the distance vector is normalized to obtain a corresponding vector, which is used to characterize the topological structure of the feature subset. The feature subset includes a first feature subset and a second feature subset; the Euclidean distance includes a first Euclidean distance and a second Euclidean distance; the distance vector includes a first distance vector and a second distance vector; and the vector includes a first vector and a second vector.

[0123] The vector determination submodule is used to sort the feature points in the feature subset according to a set order; determine the Euclidean distance between each pair of feature points; and sort the Euclidean distances according to the set order to obtain the corresponding distance vector.

[0124] The determining submodule is used to determine the cosine similarity between the first vector and the second vector.

[0125] The sorting module 608 is used to determine the similarity score between the source image and the similar image based on the similarity; and to sort the similar images in the similar image set according to their corresponding similarity scores to obtain the similar image filtering results.

[0126] The device further includes: a similar image determination module, used to acquire a source image; and to perform a search based on the content of the source image to determine a set of similar images of the source image.

[0127] The feedback module is used to filter the similar images based on the similarity score and provide feedback on the filtered similar images.

[0128] In summary, by using a source image and similar images from a set of similar images to form at least one image pair, feature matching is performed on the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset. Based on the topological structure information of the feature points, the similarity between the first feature subset and the second feature subset is determined. Similarity is verified based on the similarity of the topological structure. This method has low computational load and low resource consumption, thereby enabling the determination of similar image filtering results in the set of similar images based on the similarity, thus improving the retrieval efficiency of similar images.

[0129] Based on the above embodiments, this embodiment also provides an image search device that can be applied to electronic devices such as terminal devices and servers.

[0130] Reference Figure 7 The diagram shows a structural block diagram of an embodiment of an image search device according to this application, which may specifically include the following modules:

[0131] The request receiving module 702 is used to receive a retrieval request, which includes a source image.

[0132] The coarse ranking module 704 is used to search the graph database based on the content of the source image to determine a set of similar images of the source image, wherein the set of similar images includes at least one similar image.

[0133] The fine sorting module 706 is used to form at least one image pair by using the source image and similar images in the similar image set; to perform feature matching on the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset; to determine the similarity between the first feature subset and the second feature subset based on the topological structure information of the feature points; and to filter the similar images in the similar image set according to the similarity.

[0134] The result return module 708 is used to determine the search results based on the similar images obtained from the filtering and to send the search results.

[0135] In the context of searching for product objects, the source image is an image of a product object; the similar images include images of various product objects stored in the database of the e-commerce server.

[0136] The request receiving module 702 is used to receive a product object search request, wherein the product object search request includes an image of a product object;

[0137] The coarse ranking module 704 is used to perform retrieval based on the image of the product object and determine a set of similar images of the product object, wherein the set of similar images includes at least one similar image of a similar product object;

[0138] The fine sorting module 706 is used to form at least one image pair by using the image of the product object and similar images in the similar image set; to perform feature matching on the image of the product object and similar images in the image pair to determine the corresponding first feature subset and second feature subset; to determine the similarity between the first feature subset and the second feature subset based on the topological structure information of the feature points; and to filter the similar images in the similar image set according to the similarity.

[0139] The result return module 708 is used to determine the product object information of the corresponding similar product object based on the similar images obtained by filtering; and to send the product object information of the similar product object.

[0140] In the road search scenario, the source image is a road image; the similar images include various road surveillance images and other images collected on the road.

[0141] The request receiving module 702 is used to receive a road search request, the road search request including a road image;

[0142] The coarse sorting module 704 is used to perform retrieval based on the road image and determine a set of similar images of the road image, wherein the set of similar images includes at least one similar image of a similar product object;

[0143] The fine sorting module 706 is used to form at least one image pair by using the road image and similar images in the similar image set; to perform feature matching on the road image and similar images in the image pair to determine the corresponding first feature subset and second feature subset; to determine the similarity between the first feature subset and the second feature subset based on the topological structure information of the feature points; and to filter the similar images in the similar image set according to the similarity.

[0144] The result return module 708 is used to determine the road information corresponding to the filtered similar images and send the road information.

[0145] The background technique estimates the transformation matrix of feature points, then selects a set of point pairs S, and iteratively executes this process. This requires calculating the transformation points of all points in A under the current transformation matrix, resulting in a large computational load. In contrast, the embodiments of this application, based on a first and second feature subset, only require calculating the distance between each pair of feature points to obtain the topological structure representation vector, significantly reducing the computational load. Furthermore, the background technique requires sampling, which can lead to unstable results and discrepancies between multiple calculations. The embodiments of this application, utilizing vector similarity calculations, ensure consistent results across multiple calculations, resulting in higher accuracy.

[0146] Furthermore, the time complexity of the method described in the background technique is O(tk+tn), where t is the number of iterations, k is the average number of points involved in the transformation matrix calculation in each iteration, and n is the number of input feature point pairs. For image matching, t >> n is usually the case, and due to the matrix inversion operation involved, the constant term of the time complexity is also very large. In contrast, the time complexity of the method in the embodiment of this application is only O(n^2), which is more efficient.

[0147] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0148] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices, servers (clusters), cloud devices, and edge computing devices.

[0149] The embodiments of this disclosure can be implemented as an apparatus with any suitable hardware, firmware, software, or any combination thereof configured as desired, including electronic devices such as terminal devices, servers (clusters), cloud devices, edge computing devices, etc. Figure 8 An exemplary apparatus 800 is schematically shown that can be used to implement the various embodiments described in this application.

[0150] In one embodiment, Figure 8 An exemplary device 800 is shown, which includes one or more processors 802, a control module (chipset) 804 coupled to at least one of the processors 802, a memory 806 coupled to the control module 804, a non-volatile memory (NVM) / storage device 808 coupled to the control module 804, one or more input / output devices 810 coupled to the control module 804, and a network interface 812 coupled to the control module 804.

[0151] Processor 802 may include one or more single-core or multi-core processors, and processor 802 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 800 can serve as a terminal device, server (cluster), cloud device, edge computing device, or other device as described in the embodiments of this application.

[0152] In some embodiments, apparatus 800 may include one or more computer-readable media (e.g., memory 806 or NVM / storage device 808) having instructions 814 and one or more processors 802 that are combined with the one or more computer-readable media and configured to execute the instructions 814 to implement the module and thus perform the actions described in this disclosure.

[0153] In one embodiment, the control module 804 may include any suitable interface controller to provide any suitable interface to at least one of the processors 802 and / or any suitable device or component communicating with the control module 804.

[0154] The control module 804 may include a memory controller module to provide an interface to the memory 806. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0155] Memory 806 may be used, for example, to load and store data and / or instructions 814 for device 800. In one embodiment, memory 806 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 806 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0156] In one embodiment, the control module 804 may include one or more input / output controllers to provide an interface to the NVM / storage device 808 and (one or more) input / output devices 810.

[0157] For example, NVM / storage device 808 may be used to store data and / or instructions 814. NVM / storage device 808 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0158] NVM / storage device 808 may include storage resources that are physically part of a device on which device 800 is mounted, or that are accessible to the device but do not necessarily have to be part of the device. For example, NVM / storage device 808 may be accessed via a network via one or more input / output devices 810.

[0159] One or more input / output devices 810 may provide an interface for device 800 to communicate with any other suitable device. Input / output devices 810 may include communication components, audio components, sensor components, etc. A network interface 812 may provide an interface for device 800 to communicate via one or more networks. Device 800 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof.

[0160] In one embodiment, at least one of the processors 802 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 804. In one embodiment, at least one of the processors 802 may be logically packaged with one or more controllers of the control module 804 to form a system-in-package (SiP). In one embodiment, at least one of the processors 802 may be integrated with the logic of one or more controllers of the control module 804 on the same die. In one embodiment, at least one of the processors 802 may be integrated with the logic of one or more controllers of the control module 804 on the same die to form a system-on-a-chip (SoC).

[0161] In various embodiments, device 800 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop, handheld computing device, tablet, netbook, etc.). In various embodiments, device 800 may have more or fewer components and / or different architectures. For example, in some embodiments, device 800 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0162] The detection device can use a main control chip as a processor or control module, and sensor data, position information, etc. can be stored in a memory or NVM / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0163] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0164] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0165] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0166] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0167] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0168] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0169] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0170] The foregoing has provided a detailed description of an image processing method and apparatus, an image retrieval method and apparatus, a product object search method, a road search method, an electronic device, and a storage medium provided by this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image processing method, characterized in that, The method includes: At least one image pair is formed by using a source image and a set of similar images of the source image, wherein the set of similar images includes at least one similar image; The image pair is used to perform feature matching between the source image and similar images to determine the corresponding first feature subset and second feature subset; Determine the topological structure information of the feature points in the first feature subset to obtain the corresponding first vector; Determine the topological structure information of the feature points in the second feature subset to obtain the corresponding second vector, wherein the distance between each pair of feature points is calculated based on the coordinates of each feature point in the feature subset to obtain the corresponding vector; Determine the similarity between the first vector and the second vector; Based on the similarity, the similar image filtering results in the similar image set are determined.

2. The method according to claim 1, characterized in that, The step of performing feature matching between the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset includes: Feature extraction is performed on the source image and similar images in the image pair to determine the corresponding first feature set and second feature set; Feature point matching is performed based on the first feature set and the second feature set to determine the first feature subset and the second feature subset.

3. The method according to claim 1, characterized in that, The step of determining the topological structure information of feature points in the feature subset to obtain the corresponding vector includes: Determine the Euclidean distance between feature points in the feature subset, and determine the corresponding distance vector based on the Euclidean distance; The distance vector is normalized to obtain a corresponding vector, which is used to characterize the topological structure of the feature subset.

4. The method according to claim 3, characterized in that, The process of determining the Euclidean distance between feature points in the feature subset and determining the corresponding distance vector based on the Euclidean distance includes: Sort the feature points in the feature subset according to a set order; Determine the Euclidean distance between any two feature points; The Euclidean distances are sorted in a set order to obtain the corresponding distance vectors.

5. The method according to claim 1, characterized in that, Determining the similarity between the first vector and the second vector includes: Determine the cosine similarity between the first vector and the second vector.

6. The method according to claim 1, characterized in that, The step of determining the similar image filtering results in the similar image set based on the similarity includes: The similarity score between the source image and similar images is determined based on the similarity. The similar images in the set of similar images are sorted according to their corresponding similarity scores to obtain the similar image filtering results.

7. The method according to claim 1, characterized in that, Also includes: Get the source image; Based on the content of the source image, a set of similar images is determined.

8. The method according to claim 6, characterized in that, Also includes: The similar images are filtered based on the similarity score, and the filtered similar images are returned.

9. An image retrieval method, characterized in that, The method includes: Receive a retrieval request, the retrieval request including the source image; Based on the content of the source image, a search is performed in the graph database to determine a set of similar images of the source image, wherein the set of similar images includes at least one similar image; At least one image pair is formed by using the source image and similar images from the set of similar images; The image pair is used to perform feature matching between the source image and similar images to determine the corresponding first feature subset and second feature subset; Determine the topological structure information of the feature points in the first feature subset to obtain the corresponding first vector; Determine the topological structure information of the feature points in the second feature subset to obtain the corresponding second vector, wherein the distance between each pair of feature points is calculated based on the coordinates of each feature point in the feature subset to obtain the corresponding vector; Determine the similarity between the first vector and the second vector; Based on the similarity, similar images in the set of similar images are filtered. Based on the similar images obtained from the screening, the search results are determined and sent.

10. A method for searching for product objects, characterized in that, The method includes: Receive a product object search request, wherein the product object search request includes an image of the product object; Based on the image of the product object, a search is performed to determine a set of similar images of the product object, the set of similar images including at least one similar image of a similar product object; At least one image pair is formed by using the image of the product object and similar images from a set of similar images; The image of the commodity object in the image pair is matched with similar images to determine the corresponding first feature subset and second feature subset; Determine the topological structure information of the feature points in the first feature subset to obtain the corresponding first vector; Determine the topological structure information of the feature points in the second feature subset to obtain the corresponding second vector, wherein the distance between each pair of feature points is calculated based on the coordinates of each feature point in the feature subset to obtain the corresponding vector; Determine the similarity between the first vector and the second vector; Based on the similarity, similar images in the set of similar images are filtered. Based on the similar images obtained through filtering, determine the product object information of the corresponding similar product objects; Send the product object information of the similar product objects.

11. A road search method, characterized in that, The method includes: Receive a road search request, the road search request including road images; Based on the road image, a search is performed to determine a set of similar images of the road image, the set of similar images including at least one similar image of a similar product object; At least one image pair is formed by using the road image and similar images from the set of similar images; The road image and similar images in the image pair are matched for features to determine the corresponding first feature subset and second feature subset; Determine the topological structure information of the feature points in the first feature subset to obtain the corresponding first vector; Determine the topological structure information of the feature points in the second feature subset to obtain the corresponding second vector, wherein the distance between each pair of feature points is calculated based on the coordinates of each feature point in the feature subset to obtain the corresponding vector; Determine the similarity between the first vector and the second vector; Based on the similarity, similar images in the set of similar images are filtered. Determine the road information corresponding to the similar images obtained through filtering; Send the road information.

12. An image processing apparatus, characterized in that, include: The image pair determination module is used to form at least one image pair by using a source image and similar images from a set of similar images of the source image, wherein the set of similar images includes at least one similar image; The feature matching module is used to perform feature matching between the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset; The similarity determination module is used to determine the topological structure information of feature points in the first feature subset to obtain the corresponding first vector; determine the topological structure information of feature points in the second feature subset to obtain the corresponding second vector, wherein the distance between each pair of feature points is calculated based on the coordinates of each feature point in the feature subset to obtain the corresponding vector; and determine the similarity between the first vector and the second vector. The sorting module is used to determine the similar image filtering results in the similar image set based on the similarity.

13. An image retrieval device, characterized in that, include: A request receiving module is used to receive a retrieval request, wherein the retrieval request includes a source image; The coarse ranking module is used to search the graph database based on the content of the source image to determine a set of similar images of the source image, wherein the set of similar images includes at least one similar image; The fine-sorting module is used to form at least one image pair by using the source image and similar images from the similar image set; to perform feature matching on the source image and similar images in the image pair to determine the corresponding first feature subset and second feature subset; to determine the topological structure information of the feature points in the first feature subset to obtain the corresponding first vector; to determine the topological structure information of the feature points in the second feature subset to obtain the corresponding second vector, wherein the distance between each pair of feature points is calculated based on the coordinates of each feature point in the feature subset to obtain the corresponding vector; to determine the similarity between the first vector and the second vector; and to filter the similar images in the similar image set according to the similarity. The result return module is used to determine the search results based on the similar images obtained from the filtering, and then send the search results.

14. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method as described in one or more of claims 1-11.

15. One or more machine-readable media having executable code stored thereon, which, when executed, causes a processor to perform the method as described in one or more of claims 1-11.

Images