Image query method, device, storage medium and computer program product

Abstract

The present application discloses an image query method, device, storage medium and computer program product. The method comprises: determining a first query vector corresponding to an image feature of an image to be queried; performing a query in an image retrieval database based on the first query vector to obtain a first image set; determining a second query vector based on the first image set, and performing a query in an image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; using a classification model, determining an image in the second image set corresponding to the target class query vector, and obtaining an image query result corresponding to the image to be queried. The present application solves the technical problem that when related technologies retrieve images similar to the query image in an image database, the images provided by the queryer need to be stored in plain text on a cloud server, which may lead to the risk of personal privacy leakage.

Description

Technical Field

[0001] The present application relates to the technical field of computer image processing, and in particular to an image query method, device, storage medium, and computer program product. Background Art

[0002] Content-Based Image Retrieval (CBIR) is a mainstream image retrieval technology and a research hotspot in multimedia processing. CBIR systems are used to retrieve images similar to a query image from large image databases. However, related technologies require storing the images provided by the queryer in plaintext on a cloud server when searching for images similar to the query image. This exposes the user's search intent and poses a risk of privacy leakage.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] The embodiments of the present application provide an image query method, apparatus, storage medium, and computer program product to at least address the technical problem that, when related technologies retrieve images similar to the query image in an image database, the image provided by the queryer needs to be stored in plain text on a cloud server, which may lead to the risk of personal privacy leakage.

[0005] According to one aspect of an embodiment of the present application, an image query method is provided, including: obtaining a feature matrix corresponding to image features of an image to be queried, determining a feature vector and a feature value of the feature vector corresponding to the feature matrix, and determining a first query vector based on a feature vector whose feature value is greater than a preset feature value threshold; performing an image query in an image retrieval database based on the first query vector to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between the image features corresponding to the first query vector is greater than a preset similarity threshold; determining a second query vector based on the image features of the images in the first image set, and performing an image query in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; using a classification model to determine the image in the second image set corresponding to the target class query vector, and obtain an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

[0006] Optionally, determining the second query vector based on the image features of the images in the first image set includes: in response to the image selection instruction, determining the target image corresponding to the image selection instruction from the first image set to obtain a target class image set; determining the target class query vector based on the features of the images in the target class image set; clustering the images in the remaining image set to obtain multiple clusters, and determining the confusion class query vector based on the features of the images corresponding to the core points in the clusters, wherein the remaining image set contains the remaining images in the first image set except the target image in the target class image set; generating the second query vector based on the target class query vector and the confusion class query vector.

[0007] Optionally, clustering the images in the remaining image set includes: mapping each image in the remaining image set to a sample space, and determining a neighborhood radius and a minimum number of samples in the neighborhood, wherein, in the sample space, each image corresponds to an image sample point; selecting an unvisited image sample point for access, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and creating a cluster for the core point, and adding all image sample points in the neighborhood to the cluster, wherein the neighborhood is an area within a neighborhood radius centered on the image sample point; traversing the unvisited image sample points in the cluster, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and adding all image sample points in the neighborhood to the cluster; after traversing the image sample points in the cluster, reselecting an unvisited image sample point in the sample space for access, and repeating the above process until all image sample points in the sample space have been visited.

[0008] Optionally, the method also includes: taking the images in the target class image set as positive classes and the images corresponding to the core points in the clusters corresponding to the remaining image sets as negative classes to construct a training set; during the training process, using a support vector machine model to determine the category corresponding to the image based on the features of the image in the training set, and determining the loss function value based on the category determined by the support vector machine model and the actual category corresponding to the image in the training set, wherein the categories include: positive and negative classes; based on the loss function value, updating the model parameters of the support vector machine model, and using the support vector machine model after the updated model parameters, repeating the above training process until the loss function value is less than a preset error threshold to obtain a classification model; using the classification model, determining the category corresponding to each image in the second image set, and determining the image with the positive category as the image corresponding to the target class query vector.

[0009] Optionally, based on the first query vector, an image query is performed in the image retrieval database to obtain a first image set, including: determining an image set in the image retrieval database with the highest image feature similarity corresponding to the first query vector, wherein there are multiple image sets in the image retrieval database, and the similarity between images in the same image set is higher than a preset similarity threshold; determining the distance between the feature vector corresponding to each image in the image set and the first query vector, and determining the image corresponding to the feature vector whose distance is less than the preset distance threshold as the image in the first image set, wherein the distance is used to characterize the degree of similarity between the two images.

[0010] Optionally, determining the first query vector includes: extracting image features of the image to be queried, wherein the image features include at least one of the following: color features, texture features, and shape features; standardizing various image features, and determining a feature matrix based on the standardized image features; determining a covariance matrix corresponding to the feature matrix, and calculating eigenvectors and eigenvalues ​​corresponding to the covariance matrix; forming an eigenvector matrix from eigenvectors whose eigenvalues ​​are greater than a preset eigenvalue threshold, and performing dimensionality reduction on the eigenvector matrix to obtain the first query vector.

[0011] Optionally, extracting image features of the image to be queried includes: determining the hue component, saturation component and brightness component corresponding to the image to be queried based on the red component, green component and blue component of each pixel in the image to be queried, and determining the color features corresponding to the image to be queried based on the hue component, saturation component and brightness component; converting the image to be queried into a grayscale image, and determining the grayscale value of each pixel in the grayscale image, determining the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter corresponding to the image to be queried, and determining the texture features corresponding to the image to be queried based on the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter, wherein the smoothness parameter is used to characterize the degree of smoothness between the grayscale values ​​of each pixel in the grayscale image, the contrast parameter is used to characterize the degree of contrast of the grayscale levels in the grayscale image, the complexity parameter is used to characterize the degree of complexity of the grayscale levels in the grayscale image, and the linear correlation parameter is used to characterize the degree of linear correlation of the grayscale levels in the grayscale image; extracting the outline of the main part of the image in the image to be queried, and determining the shape features corresponding to the image to be queried based on the outline.

[0012] According to another aspect of an embodiment of the present application, an image query device is also provided, including: a feature extraction module, used to obtain a feature matrix corresponding to the image features of an image to be queried, determine the feature vectors and the feature values ​​of the feature vectors corresponding to the feature matrix, and determine a first query vector based on the feature vectors whose feature values ​​are greater than a preset feature value threshold; a first query module, used to perform an image query in an image retrieval database based on the first query vector to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between the image features corresponding to the first query vector is greater than a preset similarity threshold; a second query module, used to determine a second query vector based on the image features of the images in the first image set, and perform an image query in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; a result classification module, used to use a classification model to determine the images in the second image set corresponding to the target class query vector, and obtain an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

[0013] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided. The non-volatile storage medium includes a stored computer program, wherein a device where the non-volatile storage medium is located executes the image query method by running the computer program.

[0014] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, which implements the steps of the image query method when executed by a processor.

[0015] In an embodiment of the present application, a feature matrix corresponding to the image features of the image to be queried is obtained, the feature vectors and the feature values ​​of the feature vectors corresponding to the feature matrix are determined, and a first query vector is determined based on the feature vectors whose feature values ​​are greater than a preset feature value threshold value; based on the first query vector, an image query is performed in an image retrieval database to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between the image features corresponding to the first query vector is greater than a preset similarity threshold value; based on the image features of the images in the first image set, a second query vector is determined, and based on the second query vector, an image query is performed in the image retrieval database to obtain a second image set, wherein the second query vector includes: target class query vector and confusion class query vector; a classification model is used to determine the image corresponding to the target class query vector in the second image set, and an image query result corresponding to the image to be queried is obtained, wherein the classification model is trained based on the images corresponding to the target class query vector and the confusion class query vector in the first image set, and by using some image features selected by the user for similarity measurement, it achieves the purpose of protecting the privacy of the queryer while ensuring that the CBIR system can effectively retrieve image information, thereby solving the technical problem that when the relevant technology retrieves images similar to the query image in the image database, the image provided by the queryer needs to be stored in plain text on the cloud server, which may lead to the risk of personal privacy leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 This is a hardware structure block diagram of a computer terminal (or electronic device) for implementing an image query method provided in an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of a method flow for image query provided according to an embodiment of the present application;

[0019] Figure 3 Schematic diagram of the structure of an image query device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] To facilitate those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns involved in the embodiments of the present application are explained as follows:

[0023] CBIR (Content-Based Image Retrieval) is a technology that uses image content features to retrieve images. CBIR compares a query image with images in a database and returns the image with the highest similarity to the query image as the search result. CBIR technology has a wide range of applications in image search, image classification, and automatic image annotation.

[0024] While people enjoy the convenience of image search, they also face the risk of privacy leaks. How to ensure convenient search while protecting user privacy has become a hot topic of research. Cloud servers are untrustworthy, and when images are stored in plaintext on cloud servers, they are likely to be leaked. Therefore, image owners want to encrypt their images and upload them to cloud servers. However, previous plaintext image retrieval solutions fail in encrypted environments.

[0025] In related technologies, researchers and social media sites have developed various privacy policies and tools to protect the privacy of image owners. However, most current research results focus on encrypted images rather than the characteristics of encrypted images, which exposes users' retrieval intentions.

[0026] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0027] According to an embodiment of the present application, an embodiment of a method for image query is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG1 shows a hardware structure block diagram of a computer terminal (or electronic device) for implementing an image query method. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more (illustrated as 102a, 102b, ..., 102n in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0029] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0030] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image query method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implementing the above-mentioned image query method. The memory 104 may include a high-speed random access memory and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely located relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0031] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0032] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0033] In the above operating environment, the embodiment of the present application provides an image query method. Figure 2 is a schematic diagram of a method flow for image query provided according to an embodiment of the present application, such as Figure 2 As shown, the method includes the following steps:

[0034] Step S202, obtaining a feature matrix corresponding to the image features of the image to be queried, determining the eigenvectors and eigenvalues ​​of the eigenvectors corresponding to the feature matrix, and determining a first query vector based on the eigenvectors whose eigenvalues ​​are greater than a preset eigenvalue threshold;

[0035] Step S204: performing an image search in an image retrieval database based on the first query vector to obtain a first image set, wherein the first image set includes images in the image retrieval database whose similarity between image features corresponding to the first query vector is greater than a preset similarity threshold;

[0036] Step S206: determining a second query vector based on the image features of the images in the first image set, and performing an image query in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector;

[0037] In step S208, a classification model is used to determine the image corresponding to the target class query vector in the second image set, and an image query result corresponding to the image to be queried is obtained, wherein the classification model is trained based on the images corresponding to the target class query vector and the confusion class query vector in the first image set, respectively.

[0038] Through the above steps, by using some image features selected by the user for similarity measurement, the goal of protecting the privacy of the queryer is achieved while ensuring that the CBIR system can effectively retrieve image information. This solves the technical problem that when related technologies retrieve images similar to the query image in the image database, the images provided by the queryer need to be stored in plain text on the cloud server, which may lead to the risk of personal privacy leakage.

[0039] The image query method in steps S202 to S208 of the embodiment of the present application is further introduced below.

[0040] First, a Transformer encoder may be used to extract image features of the query image, wherein the image features include at least one of the following: color features, texture features, and shape features. The specific steps are as follows.

[0041] In some embodiments of the present application, extracting image features of the query image includes the following steps: determining the hue component, saturation component, and brightness component corresponding to the query image based on the red component, green component, and blue component of each pixel in the query image, and determining the color features corresponding to the query image based on the hue component, saturation component, and brightness component; converting the query image into a grayscale image, and determining the grayscale value of each pixel in the grayscale image, determining the smoothness parameter, contrast parameter, complexity parameter, and linear correlation parameter corresponding to the query image, and determining the texture features corresponding to the query image based on the smoothness parameter, contrast parameter, complexity parameter, and linear correlation parameter, wherein the smoothness parameter is used to characterize the degree of smoothness between the grayscale values ​​of each pixel in the grayscale image, the contrast parameter is used to characterize the degree of contrast of the grayscale levels in the grayscale image, the complexity parameter is used to characterize the complexity of the grayscale levels in the grayscale image, and the linear correlation parameter is used to characterize the degree of linear correlation of the grayscale levels in the grayscale image; extracting the contour of the main part of the image in the query image, and determining the shape features corresponding to the query image based on the contour.

[0042] The following introduces the process steps for extracting color features, texture features, and shape features respectively.

[0043] First, the specific steps of extracting color features are introduced.

[0044] Convert the image from RGB to HSV, then quantize the HSV. Normalize the R, G, and B components (i.e., the red, green, and blue components) of each pixel in the RGB space from 0-255 to 0-1. Calculate the minimum and maximum values ​​of each pixel, i.e., the minimum (min(R,G,B)) and maximum (max(R,G,B)) of the R, G, and B channels.

[0045] After that, the brightness of the pixel (Value, V) is calculated: V = max(R, G, B).

[0046] Calculate the saturation (Saturation, S) of the pixel: if the maximum and minimum values ​​are both 0, the saturation is 0; otherwise, the saturation is (max(R,G,B)-min(R,G,B)) / max(R,G,B).

[0047] Calculate the hue (Hue, H) of the pixel: If both the maximum and minimum values ​​are 0, the hue is 0; otherwise, calculate according to which of the maximum values ​​is R, G, or B: if the maximum value is R, then H = (GB) / (max(R,G,B)-min(R,G,B)); if the maximum value is G, then H = 2+(BR) / (max(R,G,B)-min(R,G,B)); if the maximum value is B, then H = 4+(RG) / (max(R,G,B)-min(R,G,B)).

[0048] Get the hue component H, saturation component S and brightness component V of the image. Finally, the above three components are combined into the color feature F corresponding to the image C , as shown below:

[0049] F C =VQ H +HQ V Q S +SQ V Q H

[0050] Among them, Q H is the quantization level of the hue component H, Q S is the quantization level of saturation S, Q V is the quantization level of brightness V.

[0051] The specific steps of extracting texture features are introduced below.

[0052] First, the input color or grayscale query image is converted to a grayscale image. Then, the region of interest or the entire image can be selected for texture feature extraction as needed. First, an offset (h, k) is defined to represent the distance moved in the image. Based on the offset, a co-occurrence matrix M(h, k) is constructed. The frequency of grayscale pairs in the image is counted. For the element m at position (h, k) in the co-occurrence matrix, hk , which represents the frequency with which a pixel with gray level i is adjacent to a pixel with gray level j at a given offset.

[0053] According to the element m in the co-occurrence matrix M(h,k) hk Calculate the texture parameters (features) F of the image S , as shown below:

[0054] F S =Σ h Σ k (m hk ) 2 +Σ h Σ k (hk) 2 +Σ h Σ k m hk logm hk +F COR

[0055] Among them, the smoothness parameter Σ h Σ k (m hk ) 2 It is used to indicate the smoothness of the image. The calculation method is to sum the squares of each element of the co-occurrence matrix.

[0056] Comparative parameter Σ h Σ k (hk) 2 It is used to represent the degree of grayscale contrast in an image. The calculation method is to multiply each element of the co-occurrence matrix by the grayscale difference of its corresponding position and then sum them.

[0057] Complexity parameter Σ h Σ k m hk logm hk It is used to represent the complexity of the grayscale in the image. The calculation method is to multiply each element of the co-occurrence matrix by the logarithm of the grayscale of its corresponding position and then sum them up.

[0058] Linear correlation parameter F CORIt is used to represent the linear correlation degree of gray levels in an image. The calculation method is to multiply each element of the co-occurrence matrix by the gray level of its corresponding position and then sum them up. The specific formula is as follows:

[0059]

[0060] in, λ=x,y; ,λ=x,y.

[0061] The following is an introduction to the process steps of extracting shape features.

[0062] In the embodiment of the present application, a shape feature extraction algorithm can be used to extract the shape feature F of the query image f(x, y) T , use edge detection algorithm or contour extraction algorithm to extract the contour of the image; calculate the shape metrics of the contour, such as the area, perimeter, roundness, stretch, etc. Use shape description algorithm (such as Hu moment, Zernike moment, Fourier descriptor, etc.) to convert the shape information of the contour into a set of feature vectors, as shown below:

[0063]

[0064] Among them, V nm (r,θ) is a set of complex polynomials defined on the unit circle in polar coordinates, n and m are polynomials V nm The order of (r,θ), n is a non-negative integer, n-|m| is an even number, and n≥|m|.

[0065] Afterwards, the first query vector used for the first query can be generated based on the obtained image features such as color features, texture features, and shape features. The specific steps are as follows.

[0066] In some embodiments of the present application, determining the first query vector includes the following steps: extracting image features of the image to be queried, wherein the image features include at least one of the following: color features, texture features, and shape features; standardizing various image features, and determining a feature matrix based on the standardized image features; determining a covariance matrix corresponding to the feature matrix, and calculating the eigenvectors and eigenvalues ​​corresponding to the covariance matrix; forming an eigenvector matrix from eigenvectors whose eigenvalues ​​are greater than a preset eigenvalue threshold, and performing dimensionality reduction on the eigenvector matrix to obtain the first query vector.

[0067] Specifically, the extracted image feature vector F C 、F S 、F TPerform standardization to make them have the same scale and range, and then form the standardized eigenvectors into a matrix X with n rows and m columns (that is, the above-mentioned feature matrix); perform zero mean processing on each row of the matrix X, subtract the mean of this row, and calculate the corresponding covariance matrix Then, the eigenvalues ​​of the covariance matrix and the corresponding eigenvectors are calculated; the eigenvectors and the corresponding image eigenvalues ​​are sorted by importance, and the user determines the threshold value k (i.e., the preset eigenvalue threshold value mentioned above). The number of features selected is limited according to the size of the threshold k, and the first k rows are taken to form the eigenvector matrix P. The data after dimensionality reduction to k dimensions is obtained, i.e., the first query vector.

[0068] After that, the first query vector can be uploaded to the CBIR system for query. The specific steps are as follows.

[0069] In some embodiments of the present application, an image query is performed in an image retrieval database based on a first query vector to obtain a first image set, including the following steps: determining an image set in the image retrieval database having the highest image feature similarity with the image feature corresponding to the first query vector, wherein there are multiple image sets in the image retrieval database, and the similarity between images in the same image set is higher than a preset similarity threshold; determining the distance between the feature vector corresponding to each image in the image set and the first query vector, and determining the image corresponding to the feature vector whose distance is less than the preset distance threshold as the image in the first image set, wherein the distance is used to characterize the degree of similarity between the two images.

[0070] Specifically, the CBIR system locates a class of images that are similar to the query image features (first query vector) uploaded by the user by querying the index table in the blockchain server. That is, it determines the image set in the image retrieval database with the highest similarity to the image features corresponding to the first query vector. It then calculates the distance between each image feature in this class and the feature of the query image, and determines the top N images with the highest similarity as the first image set and returns them to the user client.

[0071] In this embodiment, the images stored in the index table on the blockchain server side of the CBIR system are mapped to adjacent buckets using a position-sensitive hashing algorithm, with similar feature vectors mapped to these buckets. This facilitates the search for similar images for subsequent image search and matching. The position-sensitive hashing algorithm follows these steps: First, the following two conditions must be met: Assuming that points a and b (i.e., two different encrypted images) are in an m-dimensional dataset S, the hash function F() is considered sensitive. If |ab| <= R (a preset distance threshold), then Pr[F(a) = F(b)] >= p1; if |ab| >= kR, then Pr[F(a) = F(b)] <= p2, where k is an approximate ratio and p1 and p2 represent probabilities. If K > 1 and p1 > p2, if two points a and b are close together, the probability of them being projected into the same bucket using the hash function is very high; conversely, if the two points are far apart, they are projected into the same bucket with a very low probability.

[0072] It should be noted that since this step only queries based on some image features, the resulting results are less accurate, so subsequent steps are needed to improve the query accuracy, which will be further introduced below.

[0073] After receiving the query results (first image set) returned by the CBIR system for the first time, the queryer can select the target class image set from it. Then the system can automatically generate a second query vector containing obfuscated data. The specific steps are as follows.

[0074] In some embodiments of the present application, determining the second query vector based on the image features of the images in the first image set includes the following steps: in response to the image selection instruction, determining the target image corresponding to the image selection instruction from the first image set to obtain a target class image set; determining the target class query vector based on the features of the images in the target class image set; clustering the images in the remaining image set to obtain multiple clusters, and determining the confusion class query vector based on the features of the images corresponding to the core points in the clusters, wherein the remaining image set contains the remaining images in the first image set except the target image in the target class image set; generating the second query vector based on the target class query vector and the confusion class query vector.

[0075] Specifically, first, in response to the queryer's image selection instruction, the target class image set is selected from the first image set returned by the CBIR system for the first time, and then the remaining image sets in the first image set except the target class image set can be clustered using the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm. The specific steps are as follows.

[0076] In some embodiments of the present application, clustering images in the remaining image set includes the following steps: mapping each image in the remaining image set to a sample space, and determining a neighborhood radius and a minimum number of samples in the neighborhood, wherein, in the sample space, each image corresponds to an image sample point; selecting an unvisited image sample point for access, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and creating a cluster for the core point, and adding all image sample points in the neighborhood to the cluster, wherein the neighborhood is an area within a neighborhood radius centered on the image sample point; traversing the unvisited image sample points in the cluster, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and adding all image sample points in the neighborhood to the cluster; after traversing the image sample points in the cluster, reselecting an unvisited image sample point in the sample space for access, and repeating the above process until all image sample points in the sample space have been visited.

[0077] Specifically, first initialize the parameters: set the neighborhood radius ε and the minimum number of samples in the neighborhood MinPts; select an unvisited image sample point p, if the number of sample points contained in the neighborhood of p is greater than or equal to MinPts, then take p as the core point and create a new cluster; add all sample points in the neighborhood of p to the cluster; for each sample point q in the neighborhood of p, if the number of sample points contained in the neighborhood of q is greater than or equal to MinPts, then take q as the core point and add all sample points in the neighborhood of q to the cluster; if the number of sample points contained in the neighborhood of q is less than MinPts, then only add q to the cluster, and repeat this step until there are no more sample points in the cluster.

[0078] After that, a new unvisited image sample point p is selected and the above steps are repeated until all sample points have been visited. Finally, for clusters with fewer than MinPts sample points in the sample space, the sample points in that cluster are marked as noise points. For clusters with more than or equal to MinPts sample points, all sample points except the core point are considered boundary points.

[0079] Through the above steps, the remaining image set can be clustered, and the clustering results can be divided into core points, noise points and boundary points. In this embodiment, the images corresponding to the core points and / or boundary points in the clustering results can be used as candidates for the confusion class to obtain the confusion class query vector. The inquirer will upload the second query vector constructed based on the target class query vector and the confusion class query vector to the CBIR system for a second query, which can confuse the CBIR system's inference of the inquirer's search intention.

[0080] The CBIR system performs a second query and returns the second query result, returning the N most similar images to the queryer, that is, the second image set. Due to the addition of the confusion class, the query result needs to be purified, that is, a classification model is used to determine the image in the second image set that corresponds to the target class query vector, and obtain the image query result corresponding to the image to be queried.

[0081] In some embodiments of the present application, the method also includes the following steps: taking the images in the target class image set as positive classes and the images corresponding to the core points in the clusters corresponding to the remaining image sets as negative classes to construct a training set; during the training process, using a support vector machine model to determine the category corresponding to the image based on the features of the image in the training set, and determining the loss function value based on the category determined by the support vector machine model and the actual category corresponding to the image in the training set, wherein the categories include: positive and negative classes; based on the loss function value, updating the model parameters of the support vector machine model, and using the support vector machine model after the updated model parameters, repeating the above training process until the loss function value is less than a preset error threshold to obtain a classification model; using the classification model, determining the category corresponding to each image in the second image set, and determining the image with the positive category as the image corresponding to the target class query vector.

[0082] Specifically, the queryer can train a support vector machine model using the confusion class as the negative class and the target class as the positive class. During training, the system extracts image features as input, labels images of the target class as the positive class, and images of the confusion class as the negative class. The support vector machine model learns the relationship between the features and categories of these images to build a classification model. After model training is complete, the queryer inputs the final results into the support vector machine model, which reorders the results based on the previously learned relationship between features and categories. This allows images that are more similar to the target class to be ranked higher, improving the accuracy and relevance of the query results. In this way, the queryer can purify the query results returned by the CBIR system, improve the quality of the results, and better meet user needs.

[0083] This application proposal proposes an anonymization method for protecting the privacy of queries in CBIR, which directly measures similarity using some image features selected by the user. It protects the query privacy of the inquirer and ensures that their sensitive information will not be leaked. The technology uses encryption algorithms to protect query information, ensuring that the inquirer's sensitive information such as personal preferences is not leaked to unauthorized personnel or systems, further improving the privacy protection level of the query. At the same time, it allows effective image retrieval without affecting the performance of the CBIR system. By optimizing the image retrieval algorithm and introducing an efficient index structure, it ensures that while protecting the privacy of the query, it will not significantly affect the performance of the CBIR system. Through precise query processing and effective data screening, this technology can reduce query time and computing resource consumption while maintaining a high retrieval accuracy.

[0084] Furthermore, this application solution has broad applicability and can be applied to various CBIR scenarios. For example, in image search engines, users can use this technology to perform privacy-preserving image searches, thereby preventing the leakage of sensitive information. In the field of medical image retrieval, doctors and researchers can safely use this technology for disease diagnosis and image data analysis, protecting patient privacy and data security.

[0085] According to an embodiment of the present application, an embodiment of an image query device is also provided. Figure 3 FIG. 1 is a structural diagram of an image query device provided according to an embodiment of the present application. Figure 3 As shown, the device includes:

[0086] A feature extraction module 30 is configured to obtain a feature matrix corresponding to image features of the query image, determine eigenvectors and eigenvalues ​​of the eigenvectors corresponding to the feature matrix, and determine a first query vector based on eigenvectors having eigenvalues ​​greater than a preset eigenvalue threshold;

[0087] A first query module 32 is configured to perform an image query in an image retrieval database based on a first query vector to obtain a first image set, wherein the first image set includes images in the image retrieval database whose similarity between image features corresponding to the first query vector and those of the first query vector is greater than a preset similarity threshold;

[0088] A second query module 34 is configured to determine a second query vector based on image features of the images in the first image set, and perform an image query in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes a target class query vector and a confusion class query vector;

[0089] The result classification module 36 is used to use a classification model to determine the image corresponding to the target class query vector in the second image set, and obtain an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set corresponding to the target class query vector and the confusion class query vector respectively.

[0090] Optionally, determining the second query vector based on the image features of the images in the first image set includes: in response to the image selection instruction, determining the target image corresponding to the image selection instruction from the first image set to obtain a target class image set; determining the target class query vector based on the features of the images in the target class image set; clustering the images in the remaining image set to obtain multiple clusters, and determining the confusion class query vector based on the features of the images corresponding to the core points in the clusters, wherein the remaining image set contains the remaining images in the first image set except the target image in the target class image set; generating the second query vector based on the target class query vector and the confusion class query vector.

[0091] Optionally, clustering the images in the remaining image set includes: mapping each image in the remaining image set to a sample space, and determining a neighborhood radius and a minimum number of samples in the neighborhood, wherein, in the sample space, each image corresponds to an image sample point; selecting an unvisited image sample point for access, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and creating a cluster for the core point, and adding all image sample points in the neighborhood to the cluster, wherein the neighborhood is an area within a neighborhood radius centered on the image sample point; traversing the unvisited image sample points in the cluster, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and adding all image sample points in the neighborhood to the cluster; after traversing the image sample points in the cluster, reselecting an unvisited image sample point in the sample space for access, and repeating the above process until all image sample points in the sample space have been visited.

[0092] Optionally, the image query device is also used to: construct a training set by taking the images in the target class image set as positive classes and the images corresponding to the core points in the clusters corresponding to the remaining image sets as negative classes; during the training process, using a support vector machine model to determine the category corresponding to the image based on the features of the image in the training set, and determining the loss function value based on the category determined by the support vector machine model and the actual category corresponding to the image in the training set, wherein the categories include: positive and negative; updating the model parameters of the support vector machine model based on the loss function value, and repeating the above training process using the support vector machine model after the model parameters are updated until the loss function value is less than a preset error threshold to obtain a classification model; using the classification model, determining the category corresponding to each image in the second image set, and determining the image with the positive category as the image corresponding to the target class query vector.

[0093] Optionally, based on the first query vector, an image query is performed in the image retrieval database to obtain a first image set, including: determining an image set in the image retrieval database with the highest image feature similarity corresponding to the first query vector, wherein there are multiple image sets in the image retrieval database, and the similarity between images in the same image set is higher than a preset similarity threshold; determining the distance between the feature vector corresponding to each image in the image set and the first query vector, and determining the image corresponding to the feature vector whose distance is less than the preset distance threshold as the image in the first image set, wherein the distance is used to characterize the degree of similarity between the two images.

[0094] Optionally, determining the first query vector includes: extracting image features of the image to be queried, wherein the image features include at least one of the following: color features, texture features, and shape features; standardizing various image features, and determining a feature matrix based on the standardized image features; determining a covariance matrix corresponding to the feature matrix, and calculating eigenvectors and eigenvalues ​​corresponding to the covariance matrix; forming an eigenvector matrix from eigenvectors whose eigenvalues ​​are greater than a preset eigenvalue threshold, and performing dimensionality reduction on the eigenvector matrix to obtain the first query vector.

[0095] Optionally, extracting image features of the image to be queried includes: determining the hue component, saturation component and brightness component corresponding to the image to be queried based on the red component, green component and blue component of each pixel in the image to be queried, and determining the color features corresponding to the image to be queried based on the hue component, saturation component and brightness component; converting the image to be queried into a grayscale image, and determining the grayscale value of each pixel in the grayscale image, determining the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter corresponding to the image to be queried, and determining the texture features corresponding to the image to be queried based on the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter, wherein the smoothness parameter is used to characterize the degree of smoothness between the grayscale values ​​of each pixel in the grayscale image, the contrast parameter is used to characterize the degree of contrast of the grayscale levels in the grayscale image, the complexity parameter is used to characterize the degree of complexity of the grayscale levels in the grayscale image, and the linear correlation parameter is used to characterize the degree of linear correlation of the grayscale levels in the grayscale image; extracting the outline of the main part of the image in the image to be queried, and determining the shape features corresponding to the image to be queried based on the outline.

[0096] It should be noted that the various modules in the above-mentioned image query device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0097] It should be noted that the image query device provided in this embodiment can be used to perform Figure 2 The image query method shown, therefore, the relevant explanations of the above image query method are also applicable to the embodiments of the present application and will not be repeated here.

[0098] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following image query method by running the computer program: constructing a feature matrix corresponding to the image features of the image to be queried, determining the eigenvectors and eigenvalues ​​corresponding to the feature matrix, and determining a first query vector based on the eigenvector whose eigenvalue is greater than a preset eigenvalue threshold; based on the first query vector, querying in an image retrieval database to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between the image features corresponding to the first query vector is greater than a preset similarity threshold; determining a second query vector based on the first image set, and based on the second query vector, querying in the image retrieval database to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; using a classification model, determining the image in the second image set corresponding to the target class query vector, and obtaining an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

[0099] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image query method described in each embodiment of the present application: constructing a feature matrix corresponding to the image features of the image to be queried, determining the eigenvectors and eigenvalues ​​corresponding to the feature matrix, and determining a first query vector based on the eigenvector whose eigenvalue is greater than a preset eigenvalue threshold; querying in an image retrieval database based on the first query vector to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between the image features corresponding to the first query vector is greater than a preset similarity threshold; determining a second query vector based on the first image set, and querying in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; using a classification model, determining the image in the second image set corresponding to the target class query vector, and obtaining an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

[0100] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0101] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0103] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0104] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0106] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image query method, characterized in that: include: Obtaining a feature matrix corresponding to image features of the image to be queried, determining an eigenvector corresponding to the feature matrix and an eigenvalue of the eigenvector, and determining a first query vector based on the eigenvector having an eigenvalue greater than a preset eigenvalue threshold; Performing an image search in an image retrieval database based on the first query vector to obtain a first image set, wherein the first image set includes images in the image retrieval database whose similarity between image features corresponding to the first query vector and images in the image retrieval database is greater than a preset similarity threshold; Determining a second query vector based on image features of images in the first image set, and performing an image query in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; Wherein, determining the second query vector based on the image features of the images in the first image set includes: in response to an image selection instruction, determining a target image corresponding to the image selection instruction from the first image set to obtain a target class image set; determining the target class query vector based on the features of the images in the target class image set; clustering the images in the remaining image set to obtain a plurality of clusters, and determining the confusion class query vector based on the features of the images corresponding to the core points in the clusters, wherein the remaining image set includes the remaining images in the first image set except the target image in the target class image set; generating the second query vector based on the target class query vector and the confusion class query vector; A classification model is used to determine the image in the second image set corresponding to the target class query vector, and obtain an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

2. The image query method according to claim 1, characterized in that: Clustering the images in the remaining image set includes: Mapping each image in the remaining image set into a sample space, and determining a neighborhood radius and a minimum number of samples in the neighborhood, wherein in the sample space, each image corresponds to an image sample point; Selecting an unvisited image sample point for access, and if the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, determining the image sample point as a core point, creating a cluster for the core point, and adding all image sample points in the neighborhood to the cluster, wherein the neighborhood is an area within the neighborhood radius centered on the image sample point; Traversing unvisited image sample points in the cluster, and when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, determining the image sample point as a core point, and adding all image sample points in the neighborhood to the cluster; After the traversal of the image sample points in the cluster is completed, an unvisited image sample point is reselected in the sample space for access, and the above process is repeated until all image sample points in the sample space have been accessed.

3. The image query method according to claim 1, wherein: The method further comprises: The images in the target class image set are taken as positive classes, and the images corresponding to the core points in the clusters corresponding to the remaining image sets are taken as negative classes to construct a training set; During the training process, a support vector machine model is used to determine the category corresponding to the image based on the features of the image in the training set, and a loss function value is determined based on the category determined by the support vector machine model and the actual category corresponding to the image in the training set, wherein the category includes: the positive class and the negative class; According to the loss function value, the model parameters of the support vector machine model are updated, and the support vector machine model after the model parameters are updated is used to repeat the above training process until the loss function value is less than a preset error threshold, thereby obtaining the classification model; The classification model is used to determine the category corresponding to each image in the second image set, and the images belonging to the positive category are determined as images corresponding to the target category query vector.

4. The image query method according to claim 1, wherein: An image search is performed in an image retrieval database based on the first query vector, and a first image set is obtained, including: Determining, in the image retrieval database, an image set having the highest image feature similarity to the image feature corresponding to the first query vector, wherein there are multiple image sets in the image retrieval database, and the similarities between images in the same image set are all higher than a preset similarity threshold; Determine the distance between the feature vector corresponding to each image in the image set and the first query vector, and determine the image corresponding to the feature vector whose distance is less than a preset distance threshold as an image in the first image set, wherein the distance is used to represent the degree of similarity between the two images.

5. The image query method according to claim 1, wherein: Determining the first query vector includes: Extracting image features of the image to be queried, wherein the image features include at least one of the following: color features, texture features, and shape features; performing standardization processing on each type of the image features, and determining the feature matrix based on the standardized image features; Determine the covariance matrix corresponding to the characteristic matrix, and calculate the eigenvectors and eigenvalues ​​corresponding to the covariance matrix; The eigenvectors whose eigenvalues ​​are greater than the preset eigenvalue threshold are combined into an eigenvector matrix, and dimension reduction is performed on the eigenvector matrix to obtain the first query vector.

6. The image query method according to claim 5, characterized in that: Extracting the image features of the query image includes: Determining a hue component, a saturation component, and a brightness component corresponding to the query image based on the red component, the green component, and the blue component of each pixel in the query image, and determining the color feature corresponding to the query image based on the hue component, the saturation component, and the brightness component; Converting the query image into a grayscale image, determining the grayscale value of each pixel in the grayscale image, determining a smoothness parameter, a contrast parameter, a complexity parameter, and a linear correlation parameter corresponding to the query image, and determining the texture feature corresponding to the query image based on the smoothness parameter, the contrast parameter, the complexity parameter, and the linear correlation parameter, wherein the smoothness parameter is used to characterize the degree of smoothness between the grayscale values ​​of each pixel in the grayscale image, the contrast parameter is used to characterize the degree of contrast of the grayscale levels in the grayscale image, the complexity parameter is used to characterize the degree of complexity of the grayscale levels in the grayscale image, and the linear correlation parameter is used to characterize the degree of linear correlation of the grayscale levels in the grayscale image; The contour of the main part of the image to be queried is extracted, and the shape feature corresponding to the image to be queried is determined based on the contour.

7. An image query device, characterized in that: include: a feature extraction module configured to obtain a feature matrix corresponding to image features of the image to be queried, determine a feature vector corresponding to the feature matrix and an eigenvalue of the feature vector, and determine a first query vector based on the feature vector having an eigenvalue greater than a preset eigenvalue threshold; a first query module, configured to perform an image query in an image retrieval database based on the first query vector to obtain a first image set, wherein the first image set includes images in the image retrieval database whose similarity between image features corresponding to the first query vector and images is greater than a preset similarity threshold; A second query module is configured to determine a second query vector based on image features of the images in the first image set, and perform an image query in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; wherein determining the second query vector based on the image features of the images in the first image set includes: in response to an image selection instruction, determining a target image corresponding to the image selection instruction from the first image set to obtain a target class image set; determining the target class query vector based on features of the images in the target class image set; clustering the images in the remaining image set to obtain a plurality of clusters, and determining the confusion class query vector based on features of the images corresponding to core points in the clusters, wherein the remaining image set includes the remaining images in the first image set except the target image in the target class image set; and generating the second query vector based on the target class query vector and the confusion class query vector; A result classification module is used to use a classification model to determine the image corresponding to the target class query vector in the second image set, and obtain an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

8. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the image query method according to any one of claims 1 to 6 by running the computer program.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image query method according to any one of claims 1 to 6 are implemented.

Images