Image query method and device, storage medium and computer program product

Abstract

The invention discloses an image query method and device, a storage medium and a computer program product. The method comprises the following steps: determining a first query vector corresponding to an image feature of a to-be-queried image; querying in an image retrieval database according to the first query vector to obtain a first image set; according to the first image set, a second query vector is determined, query is carried out in an image retrieval database according to the second query vector, a second image set is obtained, and the second query vector comprises a target class query vector and a confusion class query vector; and determining an image corresponding to the target class query vector in the second image set by adopting the classification model to obtain an image query result corresponding to the to-be-queried image. According to the method and the device, the technical problem that the risk of personal privacy disclosure possibly exists due to the fact that the image provided by a querier needs to be stored on a cloud server in a plaintext form when the image similar to the query image is retrieved in an image database in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the technical field of computer image processing, and in particular to an image query method, device, storage medium and computer program product. Background Art

[0002] Content-Based Image Retrieval (CBIR) is a mainstream image retrieval technology and has always been a research hotspot in the field of multimedia processing. The CBIR system is used to retrieve images similar to the query image from a large image database. However, when the related technology retrieves images similar to the query image from the image database, it needs to store the images provided by the queryer in plain text on the cloud server, which will expose the user's retrieval intention and there is a technical problem of personal privacy leakage risk.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an image query method, device, storage medium and computer program product to at least solve the technical problem that when the related technology retrieves images similar to the query image in the image database, the image provided by the queryer needs to be stored in plain text on the cloud server, which may lead to the risk of personal privacy leakage.

[0005] According to one aspect of an embodiment of the present application, an image query method is provided, comprising: obtaining a feature matrix corresponding to image features of an image to be queried, determining a feature vector and a feature value of the feature vector corresponding to the feature matrix, and determining a first query vector based on a feature vector whose feature value is greater than a preset feature value threshold; performing an image query in an image retrieval database based on the first query vector to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between image features corresponding to the first query vector is greater than a preset similarity threshold; determining a second query vector based on the image features of the images in the first image set, and performing an image query in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; using a classification model to determine images in the second image set corresponding to the target class query vector, and obtaining an image query result corresponding to the image to be queried, wherein the classification model is trained based on images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

[0006] Optionally, determining the second query vector based on image features of images in the first image set includes: in response to an image selection instruction, determining a target image corresponding to the image selection instruction from the first image set to obtain a target class image set; determining a target class query vector based on features of the images in the target class image set; clustering the images in the remaining image set to obtain multiple clusters, and determining a confusion class query vector based on features of the images corresponding to core points in the clusters, wherein the remaining image set includes the remaining images in the first image set except the target images in the target class image set; and generating a second query vector based on the target class query vector and the confusion class query vector.

[0007] Optionally, clustering the images in the remaining image set includes: mapping each image in the remaining image set to a sample space, and determining a neighborhood radius and a minimum number of samples in the neighborhood, wherein in the sample space, each image corresponds to an image sample point; selecting an unvisited image sample point for access, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and creating a cluster for the core point, and adding all the image sample points in the neighborhood to the cluster, wherein the neighborhood is an area within a neighborhood radius centered on the image sample point; traversing the unvisited image sample points in the cluster, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and adding all the image sample points in the neighborhood to the cluster; after the traversal of the image sample points in the cluster is completed, reselecting an unvisited image sample point in the sample space for access, and repeating the above process until all the image sample points in the sample space have been visited.

[0008] Optionally, the method also includes: taking the images in the target class image set as positive classes and the images corresponding to the core points in the clusters corresponding to the remaining image sets as negative classes to construct a training set; during the training process, using a support vector machine model to determine the category corresponding to the image based on the features of the image in the training set, and determining the loss function value based on the category determined by the support vector machine model and the actual category corresponding to the image in the training set, wherein the categories include: positive and negative; updating the model parameters of the support vector machine model based on the loss function value, and using the support vector machine model after the model parameters are updated, repeating the above training process until the loss function value is less than a preset error threshold to obtain a classification model; using the classification model to determine the category corresponding to each image in the second image set, and determining the image with the positive category as the image corresponding to the target class query vector.

[0009] Optionally, based on the first query vector, an image query is performed in an image retrieval database to obtain a first image set, including: determining an image set in the image retrieval database with the highest image feature similarity corresponding to the first query vector, wherein there are multiple image sets in the image retrieval database, and the similarities between images in the same image set are all higher than a preset similarity threshold; determining the distance between the feature vector corresponding to each image in the image set and the first query vector, and determining the image corresponding to the feature vector whose distance is less than the preset distance threshold as an image in the first image set, wherein the distance is used to characterize the degree of similarity between two images.

[0010] Optionally, determining the first query vector includes: extracting image features of the image to be queried, wherein the image features include at least one of the following: color features, texture features, and shape features; standardizing various types of image features, and determining a feature matrix based on the standardized image features; determining a covariance matrix corresponding to the feature matrix, and calculating eigenvectors and eigenvalues ​​corresponding to the covariance matrix; forming an eigenvector matrix from eigenvectors whose eigenvalues ​​are greater than a preset eigenvalue threshold, and reducing the dimensionality of the eigenvector matrix to obtain the first query vector.

[0011] Optionally, extracting image features of the image to be queried includes: determining the hue component, saturation component and brightness component corresponding to the image to be queried based on the red component, green component and blue component of each pixel in the image to be queried, and determining the color features corresponding to the image to be queried based on the hue component, saturation component and brightness component; converting the image to be queried into a grayscale image, and determining the grayscale value of each pixel in the grayscale image, determining the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter corresponding to the image to be queried, and determining the texture features corresponding to the image to be queried based on the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter, wherein the smoothness parameter is used to characterize the degree of smoothness between the grayscale values ​​of each pixel in the grayscale image, the contrast parameter is used to characterize the degree of contrast of the grayscale in the grayscale image, the complexity parameter is used to characterize the degree of complexity of the grayscale in the grayscale image, and the linear correlation parameter is used to characterize the degree of linear correlation of the grayscale in the grayscale image; extracting the outline of the main part of the image in the image to be queried, and determining the shape features corresponding to the image to be queried based on the outline.

[0012] According to another aspect of an embodiment of the present application, an image query device is also provided, including: a feature extraction module, used to obtain a feature matrix corresponding to image features of an image to be queried, determine a feature vector and a feature value of the feature vector corresponding to the feature matrix, and determine a first query vector based on a feature vector whose feature value is greater than a preset feature value threshold; a first query module, used to perform an image query in an image retrieval database based on the first query vector to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between image features corresponding to the first query vector is greater than a preset similarity threshold; a second query module, used to determine a second query vector based on the image features of images in the first image set, and perform an image query in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; a result classification module, used to use a classification model to determine an image in the second image set corresponding to the target class query vector, and obtain an image query result corresponding to the image to be queried, wherein the classification model is trained based on images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

[0013] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, wherein the non-volatile storage medium includes a stored computer program, wherein a device where the non-volatile storage medium is located executes the image query method by running the computer program.

[0014] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the image query method are implemented.

[0015] In an embodiment of the present application, a feature matrix corresponding to the image features of the image to be queried is obtained, a feature vector and a feature value of the feature vector corresponding to the feature matrix are determined, and a first query vector is determined based on a feature vector whose feature value is greater than a preset feature value threshold; based on the first query vector, an image query is performed in an image retrieval database to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose image features corresponding to the first query vector have a similarity greater than a preset similarity threshold; based on the image features of the images in the first image set, a second query vector is determined, and based on the second query vector, an image query is performed in the image retrieval database to obtain a second image set, wherein the second query vector includes: target class query vector and confusion class query vector; using a classification model to determine the image corresponding to the target class query vector in the second image set, and obtain an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images corresponding to the target class query vector and the confusion class query vector in the first image set, and by using some image features selected by the user for similarity measurement, it is achieved while ensuring that the CBIR system can effectively retrieve image information and protecting the privacy of the inquirer, thereby solving the technical problem that when the relevant technology retrieves images similar to the query image in the image database, the image provided by the inquirer needs to be stored in plain text on the cloud server, which may lead to the risk of personal privacy leakage. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 It is a hardware structure block diagram of a computer terminal (or electronic device) for implementing an image query method provided in an embodiment of the present application;

[0018] Figure 2 is a schematic diagram of a method flow of image query provided according to an embodiment of the present application;

[0019] Figure 3 It is a structural schematic diagram of an image query device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0022] In order to facilitate those skilled in the art to better understand the embodiments of the present application, some technical terms or nouns involved in the embodiments of the present application are explained as follows:

[0023] CBIR (Content-Based Image Retrieval): is a technology that uses image content features to retrieve images. CBIR compares the input query image with the images in the database and returns the image with the highest similarity to the query image as the retrieval result. CBIR technology is widely used in image search, image classification, image automatic annotation and other fields.

[0024] While people enjoy the convenience of image search, they also bring the hidden danger of personal privacy leakage. How to ensure convenient search while protecting user privacy has become a research hotspot. The cloud server is untrustworthy. When images are stored in plain text on the cloud server, it is likely to cause image leakage. Therefore, the owner of the image wants to encrypt the image and then upload the encrypted image to the cloud server. However, the previous image plain text retrieval scheme will fail in an encrypted environment.

[0025] In the related art, researchers and social media sites have developed various privacy policies and tools to protect the privacy of image owners. However, most of the current research results focus on encrypted images rather than the characteristics of encrypted images, which exposes users' retrieval intentions.

[0026] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0027] According to an embodiment of the present application, an embodiment of a method for image query is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or electronic device) for implementing an image query method. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or electronic device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0030] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image query method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned image query method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0031] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0032] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or electronic device).

[0033] In the above operating environment, the present application embodiment provides an image query method. Figure 2 is a schematic diagram of a method flow of image query provided according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0034] Step S202, obtaining a feature matrix corresponding to the image features of the image to be queried, determining a feature vector and a feature value of the feature vector corresponding to the feature matrix, and determining a first query vector according to a feature vector whose feature value is greater than a preset feature value threshold;

[0035] Step S204, performing an image search in an image retrieval database according to the first query vector to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between image features corresponding to the first query vector is greater than a preset similarity threshold;

[0036] Step S206, determining a second query vector based on the image features of the images in the first image set, and performing an image query in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector;

[0037] Step S208, using the classification model to determine the image corresponding to the target class query vector in the second image set, and obtain the image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

[0038] Through the above steps, by using some image features selected by the user for similarity measurement, the purpose of protecting the privacy of the queryer is achieved while ensuring that the CBIR system can effectively retrieve image information. This solves the technical problem that when the relevant technology retrieves images similar to the query image in the image database, the image provided by the queryer needs to be stored in plain text on the cloud server, which may lead to the risk of personal privacy leakage.

[0039] The image query method in steps S202 to S208 of the embodiment of the present application is further introduced below.

[0040] First, the Transformer encoder can be used to extract image features of the query image, wherein the image features include at least one of the following: color features, texture features, and shape features. The specific steps are as follows.

[0041] In some embodiments of the present application, extracting image features of an image to be queried includes the following steps: determining the hue component, saturation component and brightness component corresponding to the image to be queried based on the red component, green component and blue component of each pixel in the image to be queried, and determining the color features corresponding to the image to be queried based on the hue component, saturation component and brightness component; converting the image to be queried into a grayscale image, and determining the grayscale value of each pixel in the grayscale image, determining the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter corresponding to the image to be queried, and determining the texture features corresponding to the image to be queried based on the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter, wherein the smoothness parameter is used to characterize the degree of smoothness between the grayscale values ​​of each pixel in the grayscale image, the contrast parameter is used to characterize the degree of contrast of the grayscale in the grayscale image, the complexity parameter is used to characterize the degree of complexity of the grayscale in the grayscale image, and the linear correlation parameter is used to characterize the degree of linear correlation of the grayscale in the grayscale image; extracting the contour of the main part of the image in the image to be queried, and determining the shape features corresponding to the image to be queried based on the contour.

[0042] The following introduces the process steps for extracting color features, texture features, and shape features respectively.

[0043] First, the specific steps of extracting color features are introduced.

[0044] The image is transformed from RGB color space to HSV color space, and then the HSV color space is quantized, and the value range of the R, G, and B components (i.e., the red component, green component, and blue component mentioned above) of each pixel in the RGB color space is normalized from 0-255 to 0-1. The minimum and maximum values ​​of each pixel are calculated, i.e., the minimum value (min(R,G,B)) and the maximum value (max(R,G,B)) in the R, G, and B channels.

[0045] Afterwards, the brightness of the pixel (Value, V) is calculated: V = max(R, G, B).

[0046] Calculate the saturation (Saturation, S) of the pixel: if the maximum and minimum values ​​are both 0, the saturation is 0; otherwise, the saturation is (max(R,G,B)-min(R,G,B)) / max(R,G,B).

[0047] Calculate the hue (Hue, H) of the pixel: If both the maximum and minimum values ​​are 0, the hue is 0; otherwise, calculate according to which of R, G, and B the maximum value is: if the maximum value is R, H = (GB) / (max(R,G,B)-min(R,G,B)); if the maximum value is G, H = 2+(BR) / (max(R,G,B)-min(R,G,B)); if the maximum value is B, H = 4+(RG) / (max(R,G,B)-min(R,G,B)).

[0048] Get the hue component H, saturation component S and brightness component V of the image. Finally, the above three components are combined into the color feature F corresponding to the image C , as shown below:

[0049] F C =VQ H +HQ V Q S +SQ V Q H

[0050] Among them, Q H is the quantization level of the hue component H, Q S is the quantization level of saturation S, Q V is the quantization level of brightness V.

[0051] The specific steps for extracting texture features are introduced below.

[0052] First, the input color or grayscale query image is converted to a grayscale image. Then, the region of interest or the entire image can be selected for texture feature extraction as needed. First, an offset (h, k) is defined to represent the distance moved in the image. Based on the offset, a co-occurrence matrix M(h, k) is constructed. The frequency of grayscale pairs appearing in the image is counted. For the element m at position (h, k) in the co-occurrence matrix hk , which represents the frequency with which a pixel with gray level i is adjacent to a pixel with gray level j at a given offset.

[0053] According to the element m in the co-occurrence matrix M(h,k) hk Calculate the texture parameters (features) F of the image S , as shown below:

[0054] F S =Σ h Σ k (m hk ) 2 +Σ h Σ k (hk) 2 +Σ h Σ k m hk logm hk +F COR

[0055] Among them, the smoothness parameter Σ h Σ k (m hk ) 2 It is used to indicate the smoothness of an image. The calculation method is to sum the squares of each element of the co-occurrence matrix.

[0056] Comparative parameter Σ h Σ k (hk) 2 It is used to represent the contrast of gray levels in an image. The calculation method is to multiply each element of the co-occurrence matrix by the gray level difference of its corresponding position and then sum them up.

[0057] Complexity parameter Σ h Σ k m hk logm hk It is used to represent the complexity of gray levels in an image. The calculation method is to multiply each element of the co-occurrence matrix by the logarithm of the gray level at its corresponding position and then sum them up.

[0058] Linear correlation parameter F CORIt is used to represent the linear correlation degree of gray levels in an image. The calculation method is to multiply each element of the co-occurrence matrix by the gray level product of its corresponding position and then sum them up. The specific formula is as follows:

[0059]

[0060] in, λ=x,y; ,λ=x,y.

[0061] The following is an introduction to the process steps of extracting shape features.

[0062] In the embodiment of the present application, a shape feature extraction algorithm can be used to extract the shape feature F of the query image f(x, y). T , use edge detection algorithm or contour extraction algorithm to extract the contour of the image; calculate the shape measurement of the contour, such as the area, perimeter, roundness, stretch, etc. Use shape description algorithm (such as Hu moment, Zernike moment, Fourier descriptor, etc.) to convert the shape information of the contour into a set of feature vectors, as shown in the following formula:

[0063]

[0064] Among them, V nm (r,θ) is a set of complex polynomials defined on the unit circle in polar coordinates, n and m are polynomials V nm The order of (r,θ), n is a non-negative integer, n-|m| is an even number, and n≥|m|.

[0065] Afterwards, the first query vector used for the first query can be generated based on the obtained image features such as color features, texture features and shape features. The specific steps are as follows.

[0066] In some embodiments of the present application, determining the first query vector includes the following steps: extracting image features of the image to be queried, wherein the image features include at least one of the following: color features, texture features, and shape features; standardizing various types of image features, and determining a feature matrix based on the standardized image features; determining a covariance matrix corresponding to the feature matrix, and calculating the eigenvectors and eigenvalues ​​corresponding to the covariance matrix; forming a eigenvector matrix from eigenvectors whose eigenvalues ​​are greater than a preset eigenvalue threshold, and reducing the dimensionality of the eigenvector matrix to obtain a first query vector.

[0067] Specifically, the extracted image feature vector F C 、F S 、F TStandardize them to have the same scale and range, and then form the standardized eigenvectors into a matrix X with n rows and m columns (i.e. the above-mentioned eigenmatrix); perform zero mean processing on each row of the matrix X, subtract the mean of this row, and calculate the corresponding covariance matrix Then the eigenvalues ​​of the covariance matrix and the corresponding eigenvectors are calculated; the eigenvectors and the corresponding image eigenvalues ​​are sorted according to importance, and the user determines the threshold (i.e., the preset eigenvalue threshold) k, and the number of features selected is limited according to the size of the threshold k. The first k rows are taken to form the eigenvector matrix P, and the data after dimensionality reduction to k dimensions is obtained, i.e., the first query vector.

[0068] Afterwards, the first query vector may be uploaded to the CBIR system for query. The specific steps are as follows.

[0069] In some embodiments of the present application, an image query is performed in an image retrieval database based on a first query vector to obtain a first image set, including the following steps: determining an image set in the image retrieval database having the highest image feature similarity with the image feature corresponding to the first query vector, wherein there are multiple image sets in the image retrieval database, and the similarities between images in the same image set are all higher than a preset similarity threshold; determining the distance between the feature vector corresponding to each image in the image set and the first query vector, and determining the image corresponding to the feature vector whose distance is less than the preset distance threshold as an image in the first image set, wherein the distance is used to characterize the degree of similarity between two images.

[0070] Specifically, the CBIR system locates a class of images that are similar to the query image features (first query vector) uploaded by the user by querying the index table in the blockchain server, that is, determines the image set with the highest similarity to the image features corresponding to the first query vector in the image retrieval database, and then calculates the distance between each image feature in this class of images and the feature of the query image, and determines the top N images with the highest similarity as the first image set and returns them to the user client.

[0071] In this embodiment, each image stored in the index table in the blockchain server of the above CBIR system maps similar feature vectors to adjacent buckets according to the position-sensitive hashing algorithm, so as to facilitate the search of similar images for subsequent image search and matching. Among them, the steps of the position-sensitive hashing algorithm are as follows: First, the following two conditions need to be met: Assuming that points a and b (i.e., two different encrypted images) are in the m-dimensional data set S, it can be said that the hash function F() is sensitive. If |ab|<=R (preset distance threshold), then Pr[F(a)=F(b)]>=p1, if |ab|>=kR, then Pr[F(a)=F(b)]<=p2, where k is an approximate ratio, and p1 and p2 represent probabilities. K>1 and p1>p2, if two points a and b are very close to each other, the probability of being projected into the same bucket by the hash function is very high; conversely, if the two points are far away from each other, they are projected into one bucket with a very low probability.

[0072] It should be noted that since this step only queries based on some image features, the resulting results are less accurate, so subsequent steps are needed to improve the query accuracy, which will be further introduced below.

[0073] After receiving the query result (first image set) returned by the CBIR system for the first time, the queryer can select the target class image set from it, and then the system can automatically generate a second query vector containing obfuscated data. The specific steps are as follows.

[0074] In some embodiments of the present application, determining the second query vector based on image features of images in the first image set includes the following steps: in response to an image selection instruction, determining a target image corresponding to the image selection instruction from the first image set to obtain a target class image set; determining a target class query vector based on features of the images in the target class image set; clustering the images in the remaining image set to obtain multiple clustering clusters, and determining a confusion class query vector based on features of the images corresponding to core points in the clustering clusters, wherein the remaining image set includes the remaining images in the first image set except the target image in the target class image set; generating a second query vector based on the target class query vector and the confusion class query vector.

[0075] Specifically, first, in response to the queryer's image selection instruction, a target class image set is selected from the first image set returned by the CBIR system for the first time, and then the remaining image sets in the first image set except the target class image set can be clustered by the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm. The specific steps are as follows.

[0076] In some embodiments of the present application, clustering the images in the remaining image set includes the following steps: mapping each image in the remaining image set to a sample space, and determining a neighborhood radius and a minimum number of samples in the neighborhood, wherein in the sample space, each image corresponds to an image sample point; selecting an unvisited image sample point for access, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and creating a cluster for the core point, and adding all the image sample points in the neighborhood to the cluster, wherein the neighborhood is an area within a neighborhood radius centered on the image sample point; traversing the unvisited image sample points in the cluster, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and adding all the image sample points in the neighborhood to the cluster; after the traversal of the image sample points in the cluster is completed, reselecting an unvisited image sample point in the sample space for access, and repeating the above process until all the image sample points in the sample space have been visited.

[0077] Specifically, first initialize the parameters: set the neighborhood radius ε and the minimum number of samples in the neighborhood MinPts; select an unvisited image sample point p, if the number of sample points contained in the neighborhood of p is greater than or equal to MinPts, then take p as the core point and create a new cluster; add all sample points in the neighborhood of p to the cluster; for each sample point q in the neighborhood of p, if the number of sample points contained in the neighborhood of q is greater than or equal to MinPts, then take q as the core point and add all sample points in the neighborhood of q to the cluster; if the number of sample points contained in the neighborhood of q is less than MinPts, then only add q to the cluster, and repeat this step until there are no more sample points in the cluster.

[0078] After that, reselect an unvisited image sample point p and repeat the above steps until all sample points are visited. Finally, for clusters whose number of sample points in the sample space is less than MinPts, the sample points in the cluster are marked as noise points. For clusters whose number of sample points is greater than or equal to MinPts, the remaining sample points except the core points are boundary points.

[0079] Through the above steps, the remaining image sets can be clustered, and the clustering results can be divided into core points, noise points and boundary points. In this embodiment, the images corresponding to the core points and / or boundary points in the clustering results can be used as candidates for confusion classes to obtain confusion class query vectors. The inquirer will upload the second query vector constructed based on the target class query vector and the confusion class query vector to the CBIR system for a second query, which can confuse the CBIR system's inference of the inquirer's search intent.

[0080] The CBIR system performs a second query and returns the second query result, returning the N most similar images to the queryer, that is, the second image set. Due to the addition of the confusion class, the query result needs to be purified, that is, the classification model is used to determine the image in the second image set that corresponds to the target class query vector, and obtain the image query result corresponding to the image to be queried.

[0081] In some embodiments of the present application, the method also includes the following steps: taking the images in the target class image set as the positive class and the images corresponding to the core points in the clusters corresponding to the remaining image sets as the negative class to construct a training set; during the training process, using a support vector machine model to determine the category corresponding to the image based on the features of the image in the training set, and determining the loss function value based on the category determined by the support vector machine model and the actual category corresponding to the image in the training set, wherein the categories include: positive and negative; based on the loss function value, updating the model parameters of the support vector machine model, and using the support vector machine model after the model parameters are updated, repeating the above training process until the loss function value is less than a preset error threshold to obtain a classification model; using the classification model to determine the category corresponding to each image in the second image set, and determining the image with the positive category as the image corresponding to the target class query vector.

[0082] Specifically, the queryer can use the confusion class as the negative class and the target class as the positive class to train the support vector machine model. During training, the system will extract the features of the image as input, mark the target class images as the positive class, and mark the confusion class images as the negative class. The support vector machine model builds a classification model by learning the relationship between the features and categories of these images. After the model training is completed, the queryer inputs the final result into the support vector machine model, and the model will reorder the results according to the previously learned relationship between the features and categories. This allows images that are more similar to the target class to be ranked first, improving the accuracy and relevance of the query results. In this way, the queryer can purify the query results returned by the CBIR system, improve the quality of the results, and better meet the needs of users.

[0083] This application scheme proposes an anonymization method for protecting the privacy of queries in CBIR, which directly measures similarity using partial image features selected by the user. The query privacy of the queryer is protected to ensure that their sensitive information will not be leaked. The technology uses encryption algorithms to protect query information to ensure that the queryer's sensitive information such as personal preferences is not leaked to unauthorized personnel or systems, further improving the privacy protection level of the query. At the same time, it allows effective image retrieval without affecting the performance of the CBIR system. By optimizing the image retrieval algorithm and introducing an efficient index structure, it ensures that while protecting the privacy of the query, it will not significantly affect the performance of the CBIR system. Through precise query processing and effective data screening, this technology can reduce query time and computing resource consumption while maintaining a high retrieval accuracy.

[0084] In addition, the application scheme has wide applicability and can be applied to various CBIR application scenarios. For example, in image search engines, users can use this technology to perform privacy-protected image searches, thereby avoiding the leakage of sensitive information. In the field of medical image retrieval, doctors and researchers can safely use this technology for disease diagnosis and image data analysis, protecting patient privacy and data security.

[0085] According to an embodiment of the present application, an embodiment of an image query device is also provided. Figure 3 is a schematic diagram of the structure of an image query device provided according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0086] The feature extraction module 30 is used to obtain a feature matrix corresponding to the image features of the image to be queried, determine the feature vectors and the feature values ​​of the feature vectors corresponding to the feature matrix, and determine the first query vector according to the feature vectors whose feature values ​​are greater than a preset feature value threshold;

[0087] A first query module 32 is used to perform an image query in an image retrieval database according to a first query vector to obtain a first image set, wherein the first image set includes images in the image retrieval database whose similarity between image features corresponding to the first query vector is greater than a preset similarity threshold;

[0088] A second query module 34 is used to determine a second query vector according to image features of the images in the first image set, and perform image query in the image retrieval database according to the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector;

[0089] The result classification module 36 is used to use a classification model to determine the image corresponding to the target class query vector in the second image set, and obtain an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

[0090] Optionally, determining the second query vector based on image features of images in the first image set includes: in response to an image selection instruction, determining a target image corresponding to the image selection instruction from the first image set to obtain a target class image set; determining a target class query vector based on features of the images in the target class image set; clustering the images in the remaining image set to obtain multiple clusters, and determining a confusion class query vector based on features of the images corresponding to core points in the clusters, wherein the remaining image set includes the remaining images in the first image set except the target images in the target class image set; and generating a second query vector based on the target class query vector and the confusion class query vector.

[0091] Optionally, clustering the images in the remaining image set includes: mapping each image in the remaining image set to a sample space, and determining a neighborhood radius and a minimum number of samples in the neighborhood, wherein in the sample space, each image corresponds to an image sample point; selecting an unvisited image sample point for access, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and creating a cluster for the core point, and adding all the image sample points in the neighborhood to the cluster, wherein the neighborhood is an area within a neighborhood radius centered on the image sample point; traversing the unvisited image sample points in the cluster, and determining the image sample point as a core point when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, and adding all the image sample points in the neighborhood to the cluster; after the traversal of the image sample points in the cluster is completed, reselecting an unvisited image sample point in the sample space for access, and repeating the above process until all the image sample points in the sample space have been visited.

[0092] Optionally, the image query device is also used to: construct a training set by taking the images in the target class image set as positive classes and the images corresponding to the core points in the clusters corresponding to the remaining image sets as negative classes; during the training process, using a support vector machine model to determine the category corresponding to the image based on the features of the image in the training set, and determining the loss function value based on the category determined by the support vector machine model and the actual category corresponding to the image in the training set, wherein the categories include: positive and negative; updating the model parameters of the support vector machine model based on the loss function value, and using the support vector machine model after the model parameters are updated, repeating the above training process until the loss function value is less than a preset error threshold, thereby obtaining a classification model; using the classification model, determining the category corresponding to each image in the second image set, and determining the image with the positive category as the image corresponding to the target class query vector.

[0093] Optionally, based on the first query vector, an image query is performed in an image retrieval database to obtain a first image set, including: determining an image set in the image retrieval database with the highest image feature similarity corresponding to the first query vector, wherein there are multiple image sets in the image retrieval database, and the similarities between images in the same image set are all higher than a preset similarity threshold; determining the distance between the feature vector corresponding to each image in the image set and the first query vector, and determining the image corresponding to the feature vector whose distance is less than the preset distance threshold as an image in the first image set, wherein the distance is used to characterize the degree of similarity between two images.

[0094] Optionally, determining the first query vector includes: extracting image features of the image to be queried, wherein the image features include at least one of the following: color features, texture features, and shape features; standardizing various types of image features, and determining a feature matrix based on the standardized image features; determining a covariance matrix corresponding to the feature matrix, and calculating eigenvectors and eigenvalues ​​corresponding to the covariance matrix; forming an eigenvector matrix from eigenvectors whose eigenvalues ​​are greater than a preset eigenvalue threshold, and reducing the dimensionality of the eigenvector matrix to obtain the first query vector.

[0095] Optionally, extracting image features of the image to be queried includes: determining the hue component, saturation component and brightness component corresponding to the image to be queried based on the red component, green component and blue component of each pixel in the image to be queried, and determining the color features corresponding to the image to be queried based on the hue component, saturation component and brightness component; converting the image to be queried into a grayscale image, and determining the grayscale value of each pixel in the grayscale image, determining the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter corresponding to the image to be queried, and determining the texture features corresponding to the image to be queried based on the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter, wherein the smoothness parameter is used to characterize the degree of smoothness between the grayscale values ​​of each pixel in the grayscale image, the contrast parameter is used to characterize the degree of contrast of the grayscale in the grayscale image, the complexity parameter is used to characterize the degree of complexity of the grayscale in the grayscale image, and the linear correlation parameter is used to characterize the degree of linear correlation of the grayscale in the grayscale image; extracting the outline of the main part of the image in the image to be queried, and determining the shape features corresponding to the image to be queried based on the outline.

[0096] It should be noted that the various modules in the above-mentioned image query device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.

[0097] It should be noted that the image query device provided in this embodiment can be used to execute Figure 2 The image query method shown, therefore, the relevant explanations and descriptions of the above-mentioned image query method are also applicable to the embodiments of the present application and will not be repeated here.

[0098] An embodiment of the present application also provides a non-volatile storage medium, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the following image query method by running the computer program: constructing a feature matrix corresponding to the image features of the image to be queried, determining the feature vectors and eigenvalues ​​corresponding to the feature matrix, and determining a first query vector based on the feature vector whose eigenvalue is greater than a preset eigenvalue threshold; querying in an image retrieval database based on the first query vector to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between the image features corresponding to the first query vector is greater than a preset similarity threshold; determining a second query vector based on the first image set, and querying in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; using a classification model, determining the image in the second image set corresponding to the target class query vector, and obtaining an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

[0099] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the image query method described in each embodiment of the present application: constructing a feature matrix corresponding to the image features of the image to be queried, determining the feature vectors and feature values ​​corresponding to the feature matrix, and determining a first query vector based on the feature vector whose feature value is greater than a preset feature value threshold; querying in an image retrieval database based on the first query vector to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between the image features corresponding to the first query vector is greater than a preset similarity threshold; determining a second query vector based on the first image set, and querying in the image retrieval database based on the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; using a classification model, determining the image in the second image set corresponding to the target class query vector, and obtaining an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

[0100] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0101] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0102] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0106] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image query method, characterized in that: include: Acquire a feature matrix corresponding to the image features of the image to be queried, determine a feature vector corresponding to the feature matrix and an eigenvalue of the feature vector, and determine a first query vector according to the eigenvector whose eigenvalue is greater than a preset eigenvalue threshold; Performing an image search in an image retrieval database according to the first query vector to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between image features corresponding to the first query vector and images is greater than a preset similarity threshold; Determine a second query vector according to image features of the images in the first image set, and perform an image query in the image retrieval database according to the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; A classification model is used to determine the image corresponding to the target class query vector in the second image set, and an image query result corresponding to the image to be queried is obtained, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

2. The image query method according to claim 1, characterized in that: Determining a second query vector according to image features of the images in the first image set includes: In response to an image selection instruction, determining a target image corresponding to the image selection instruction from the first image set to obtain a target class image set; Determining the target class query vector according to features of the images in the target class image set; Clustering the images in the remaining image set to obtain a plurality of clusters, and determining the confusion class query vector according to the features of the images corresponding to the core points in the clusters, wherein the remaining image set includes the remaining images in the first image set except the target image in the target class image set; The second query vector is generated according to the target class query vector and the confusion class query vector.

3. The image query method according to claim 2, characterized in that: Clustering the images in the remaining image set includes: Mapping each image in the remaining image set to a sample space, and determining a neighborhood radius and a minimum number of samples in the neighborhood, wherein in the sample space, each image corresponds to an image sample point; Select an unvisited image sample point for access, and when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, determine the image sample point as a core point, create a cluster for the core point, and add all image sample points in the neighborhood to the cluster, wherein the neighborhood is an area within the neighborhood radius centered on the image sample point; Traversing the unvisited image sample points in the cluster, and when the number of image sample points in the neighborhood of the image sample point is not less than the minimum number of samples, determining the image sample point as a core point, and adding all the image sample points in the neighborhood to the cluster; After the traversal of the image sample points in the cluster is completed, an unvisited image sample point is reselected in the sample space for access, and the above process is repeated until all the image sample points in the sample space have been visited.

4. The image query method according to claim 2, characterized in that: The method further comprises: The images in the target class image set are taken as positive classes, and the images corresponding to the core points in the clusters corresponding to the remaining image set are taken as negative classes to construct a training set; During the training process, a support vector machine model is used to determine the category corresponding to the image according to the features of the image in the training set, and a loss function value is determined according to the category determined by the support vector machine model and the real category corresponding to the image in the training set, wherein the category includes: the positive category and the negative category; According to the loss function value, the model parameters of the support vector machine model are updated, and the support vector machine model after the model parameters are updated is used to repeat the above training process until the loss function value is less than a preset error threshold, thereby obtaining the classification model; The classification model is used to determine the category corresponding to each image in the second image set, and the image belonging to the positive category is determined as the image corresponding to the target class query vector.

5. The image query method according to claim 1, characterized in that: According to the first query vector, an image query is performed in an image retrieval database to obtain a first image set including: Determine an image set in the image retrieval database that has the highest image feature similarity to the image feature corresponding to the first query vector, wherein there are multiple image sets in the image retrieval database, and the similarities between images in the same image set are all higher than a preset similarity threshold; Determine the distance between the feature vector corresponding to each image in the image set and the first query vector, and determine the image corresponding to the feature vector whose distance is less than a preset distance threshold as an image in the first image set, wherein the distance is used to characterize the similarity between two images.

6. The image query method according to claim 1, characterized in that: Determining the first query vector includes: Extracting image features of the image to be queried, wherein the image features include at least one of the following: color features, texture features, and shape features; Performing standardization processing on each type of the image features, and determining the feature matrix based on the standardized image features; Determine the covariance matrix corresponding to the characteristic matrix, and calculate the eigenvectors and eigenvalues ​​corresponding to the covariance matrix; The eigenvectors whose eigenvalues ​​are greater than the preset eigenvalue threshold are combined into an eigenvector matrix, and the eigenvector matrix is ​​reduced in dimension to obtain the first query vector.

7. The image query method according to claim 6, characterized in that: Extracting the image features of the image to be queried includes: Determine the hue component, saturation component and brightness component corresponding to the image to be queried according to the red component, green component and blue component of each pixel in the image to be queried, and determine the color feature corresponding to the image to be queried according to the hue component, the saturation component and the brightness component; Convert the image to be queried into a grayscale image, determine the grayscale value of each pixel in the grayscale image, determine the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter corresponding to the image to be queried, and determine the texture feature corresponding to the image to be queried based on the smoothness parameter, contrast parameter, complexity parameter and linear correlation parameter, wherein the smoothness parameter is used to characterize the smoothness between the grayscale values ​​of each pixel in the grayscale image, the contrast parameter is used to characterize the contrast degree of the grayscale in the grayscale image, the complexity parameter is used to characterize the complexity of the grayscale in the grayscale image, and the linear correlation parameter is used to characterize the linear correlation degree of the grayscale in the grayscale image; The contour of the main part of the image to be queried is extracted, and the shape feature corresponding to the image to be queried is determined based on the contour.

8. An image query device, characterized in that: include: A feature extraction module, used to obtain a feature matrix corresponding to the image features of the image to be queried, determine a feature vector corresponding to the feature matrix and an eigenvalue of the feature vector, and determine a first query vector according to the feature vector whose eigenvalue is greater than a preset eigenvalue threshold; A first query module is used to perform an image query in an image retrieval database according to the first query vector to obtain a first image set, wherein the first image set includes: images in the image retrieval database whose similarity between image features corresponding to the first query vector and images is greater than a preset similarity threshold; A second query module is used to determine a second query vector according to image features of the images in the first image set, and perform an image query in the image retrieval database according to the second query vector to obtain a second image set, wherein the second query vector includes: a target class query vector and a confusion class query vector; A result classification module is used to use a classification model to determine the image corresponding to the target class query vector in the second image set, and obtain an image query result corresponding to the image to be queried, wherein the classification model is trained based on the images in the first image set that correspond to the target class query vector and the confusion class query vector, respectively.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the image query method according to any one of claims 1 to 7 by running the computer program.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image query method according to any one of claims 1 to 7 are implemented.

Images