Online privacy protection method and device for retrieval model and storage medium

Abstract

The application provides an online privacy protection method and device for a retrieval model and a storage medium. The online privacy protection method for the retrieval model comprises: obtaining a first similarity matrix; the first similarity matrix refers to a semantic similarity matrix between image samples in a query image set; the query image set comprises a plurality of image samples; and interference of the query image set is determined based on the first similarity matrix. The online privacy protection method, device and storage medium for the retrieval model provided by the application guide the generation of interference based on the semantic similarity between image samples to provide reliable image protection, bypass the detection and identification of the retrieval model, and improve the security of information.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an online privacy protection method, apparatus, and storage medium for retrieval models. Background Technology

[0002] The rapid development of internet and social networking technologies has spurred enormous demand for online search, rendering traditional keyword-based search engines insufficient to meet the complex retrieval needs of internet users. Conversely, multimodal retrieval systems that utilize machine learning architectures, building upon existing samples to search for semantically relevant multimodal information, are more competitive in the modern internet environment.

[0003] Currently, deep hashing models, which combine hash code encoding and deep neural networks, can automatically explore semantically relevant samples from massive online data. However, this retrieval method can also retrieve some private information without clear labels, causing privacy leaks. Summary of the Invention

[0004] This invention provides an online privacy protection method, apparatus, and storage medium for retrieval models, in order to solve the technical problem that retrieval models in the prior art can lead to privacy leaks.

[0005] In a first aspect, the present invention provides an online privacy protection method for a retrieval model, comprising the following steps.

[0006] Obtain the first similarity matrix; the first similarity matrix refers to the semantic similarity matrix between image samples in the query image set; the query image set contains multiple image samples;

[0007] The interference in the query image set is determined based on the first similarity matrix.

[0008] In some embodiments, before determining the interference of the query image set based on the first similarity matrix, the method further includes:

[0009] Retrieve target text samples from a text retrieval set; the text retrieval set contains multiple text samples.

[0010] A second similarity matrix is ​​determined based on the acquired target text samples; the second similarity matrix refers to the semantic similarity matrix between the query image set and the target text samples.

[0011] In some embodiments, the method further includes:

[0012] The hash code of the target text sample is obtained using a retrieval model, and the hash code of the image in the query image set after perturbation is determined; the perturbation is within a preset range.

[0013] In some embodiments, determining the interference of the query image set based on the first similarity matrix includes:

[0014] The perturbation of the query image set is determined based on the first similarity matrix, the second similarity matrix, the hash code of the target text sample, and the hash code of the images in the query image set after perturbation.

[0015] In some embodiments, the method further includes:

[0016] Determine a first Hamming distance and a second Hamming distance; the first Hamming distance is the Hamming distance between the hash code of the image in the query image set after perturbation and the hash code of the first text sample, and the second Hamming distance is the Hamming distance between the hash code of the image in the query image set and the hash code of the second text sample; the first text sample and the second text sample are different text samples in the target text sample;

[0017] If the first Hamming distance is greater than the second Hamming distance, the process of re-acquiring target text samples and iteratively determining the interference of the query image set continues until the first Hamming distance is less than or equal to the second Hamming distance.

[0018] In some embodiments, obtaining the first similarity matrix includes:

[0019] Based on the similarity matrix between the image sample and the text sample, extract multiple image samples that are similar to the same text sample, and obtain a first similarity matrix based on the multiple image samples that are similar to the same text sample.

[0020] In some embodiments, obtaining the first similarity matrix includes:

[0021] The retrieval model is used to retrieve multiple image samples corresponding to each text sample, and the first similarity matrix is ​​obtained through the multiple image samples corresponding to each text sample.

[0022] Secondly, the present invention provides an online privacy protection device for a retrieval model, comprising the following modules.

[0023] The acquisition module is used to acquire a first similarity matrix; the first similarity matrix refers to the semantic similarity matrix between image samples in the query image set; the query image set contains multiple image samples;

[0024] A determination module is used to determine the interference of the query image set based on the first similarity matrix.

[0025] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the online privacy protection method for a retrieval model as described in the first aspect above.

[0026] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the online privacy protection method for the retrieval model as described in the first aspect above.

[0027] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the online privacy protection method for a retrieval model as described in the first aspect above.

[0028] The present invention provides an online privacy protection method, apparatus, and storage medium for retrieval models. First, a first similarity matrix is ​​obtained, which refers to the semantic similarity matrix between image samples in a query image set, which contains multiple image samples. Then, interference in the query image set is determined based on the obtained first similarity matrix. By using the semantic similarity between image samples to guide the generation of interference, reliable image protection is provided. This bypasses the detection and recognition of the retrieval model without affecting user image recognition, thus improving information security. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is one of the flowcharts illustrating the online privacy protection method for retrieval models provided by the present invention.

[0031] Figure 2 This is the second flowchart illustrating the online privacy protection method for retrieval models provided by this invention.

[0032] Figure 3 This is a schematic diagram of the online privacy protection device for retrieval models provided by the present invention.

[0033] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0034] Current multimodal retrieval models attempt to find a joint semantic subspace for various modalities. The similarity of corresponding samples can be measured by the Euclidean distance of the mappings of multimodal samples in the semantic subspace. Encoding the learned mappings with binary hash codes can improve the computational efficiency of similarity measurement and reduce storage costs. Furthermore, using deep neural networks as a formal feature extraction structure, the deep hashing framework can further improve the accuracy of retrieval models on various multimodal retrieval tasks. The use of deep neural networks also enables the model to automatically learn semantics and semantic relevance, thereby improving the accuracy of the retrieval model. Therefore, the effectiveness and generalization ability of deep hashing retrieval models are improved, enabling automated and efficient retrieval execution on large-scale websites.

[0035] On the other hand, the enhanced generalization ability of deep hashing models enables them to automatically explore semantically relevant samples from massive online datasets; however, this also raises privacy concerns on social networking sites. Some private information without clear labels may be captured by malicious crawlers designed using deep hashing retrieval models. Typically, social network users tend to use specific processing techniques to hide their private information, such as personal photos, and these processing mechanisms do not affect the ability of real users to identify processed samples.

[0036] To address the aforementioned technical issues, this invention proposes an online privacy protection method for retrieval models. First, a first similarity matrix is ​​obtained, which refers to the semantic similarity matrix between image samples in a query image set, which contains multiple image samples. Then, interference in the query image set is determined based on the obtained first similarity matrix. By using the semantic similarity between image samples to guide the generation of interference, reliable image protection is provided. This bypasses the detection and recognition of the retrieval model without affecting user image recognition, thus improving information security.

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0038] Figure 1 This is one of the flowcharts illustrating the online privacy protection method for retrieval models provided by the present invention, such as... Figure 1 As shown, this invention provides an online privacy protection method for a retrieval model. The method includes the following steps 101 and 102.

[0039] Step 101: Obtain the first similarity matrix; the first similarity matrix refers to the semantic similarity matrix between image samples in the query image set; the query image set contains multiple image samples;

[0040] Specifically, for an image-text retrieval task, given a query image set and a text retrieval set, a retrieval model is used for retrieval, which can be a powerful deep hash retrieval model.

[0041] To protect users' privacy information from being detected / identified by the retrieval model, this application proposes a hashing fake framework. It utilizes the adversarial vulnerability property of machine learning to generate perturbations (or adversarial disturbances) for images to circumvent powerful deep neural network-based retrieval models. As long as the perturbations carefully created on the image are small enough, they will not arouse human perception and therefore will not affect the recognition of human users.

[0042] Adversarial vulnerability refers to a specific phenomenon where machine learning architectures are found to be vulnerable to a set of attacks called adversarial perturbations. Adversarial perturbations are defined as a set of attacks added to samples to mislead the target model. The magnitude of these attacks is strictly limited, small enough to be imperceptible to humans. Therefore, from a human perspective, the attacked sample remains unchanged, but can mislead deep neural network models into making incorrect predictions with a high probability. Adversarial vulnerability also provides a viable solution for developing systems requiring security. Specifically, by adding signals that are imperceptible to humans to user-uploaded private data, conventional deep neural network detectors or recognizers can be effectively circumvented without affecting human users' recognition.

[0043] In this embodiment of the application, a first similarity matrix is ​​obtained, which refers to the semantic similarity matrix between image samples in the query image set.

[0044] For example, given a query image set and a text retrieval set, a retrieval model, and a similarity matrix between images in the query image set and text in the text retrieval set. Identifying images with semantic similarity can be done specifically based on the similarity matrix between images and text. To determine this, one can also retrieve which images are mapped to the same text, and then construct a similarity matrix between images in the query image set, i.e., the first matrix.

[0045] Step 102: Determine the interference of the query image set based on the first similarity matrix.

[0046] Specifically, determining the interference in the query image set involves adding interference or perturbation to each image in the query image set. In this embodiment, the semantic similarity between images in the query image set is considered, and interference in the query image set is generated based on a first similarity matrix, effectively producing perturbations that are imperceptible to the user.

[0047] The online privacy protection method for retrieval models provided in this application embodiment is based on the principle of hash forgery, which effectively hides user information. It uses the semantic similarity between images as an auxiliary supervision signal to help generate image interference, thereby misleading the identification and information extraction of malicious retrieval models and providing reliable image protection.

[0048] In some embodiments, obtaining the first similarity matrix includes:

[0049] Based on the similarity matrix between the image sample and the text sample, extract multiple image samples that are similar to the same text sample, and obtain a first similarity matrix based on the multiple image samples that are similar to the same text sample.

[0050] Specifically, the semantic similarity matrix between image samples, i.e. the first similarity matrix, can be obtained by extracting multiple images similar to the same text sample based on the similarity matrix between the image sample and the text sample, thereby constructing the similarity matrix between the images.

[0051] For example, in the similarity matrix between image and text samples, an element with a value of 1 indicates that the corresponding image sample and text sample match or are semantically similar, while an element with a value of 0 indicates that the corresponding image sample and text sample do not match or have no semantic similarity. Suppose that based on the similarity matrix between image and text samples, we extract: image samples similar to text sample 1 include image samples 2, 5, and 7; image samples similar to text sample 2 include image samples 1, 3, and 6; and image samples similar to text sample 3 include image samples 4 and 8. Then we know that image samples 2, 5, and 7 are semantically similar; image samples 1, 3, and 6 are semantically similar; and image samples 4 and 8 are semantically similar. The element corresponding to each pair of semantically similar image samples is 1, thus obtaining the similarity matrix between image samples.

[0052] In some embodiments, obtaining the first similarity matrix includes:

[0053] The retrieval model is used to retrieve multiple image samples corresponding to each text sample, and the first similarity matrix is ​​obtained through the multiple image samples corresponding to each text sample.

[0054] Specifically, the semantic similarity matrix between image samples, i.e., the first similarity matrix, can be obtained by using a retrieval model. Based on the semantic similarity between multiple image samples retrieved from a certain text sample, the first similarity matrix can be obtained from the retrieval results.

[0055] For example, using a retrieval model, image samples matching text sample 1 include image sample 2, image sample 5, and image sample 7; image samples matching text sample 2 include image sample 1, image sample 3, and image sample 6; and image samples similar to text sample 3 include image sample 4 and image sample 8. Therefore, image samples 2, 5, and 7 have semantic similarity; image samples 1, 3, and 6 have semantic similarity; and image samples 4 and 8 have semantic similarity. Since the element value corresponding to the semantically similar image sample pairs in the matrix is ​​1, a similarity matrix between image samples can be obtained.

[0056] The online privacy protection method for retrieval models provided in this application can determine the semantic similarity between image samples in multiple ways, enhancing the flexibility of the application and meeting the needs of different scenarios.

[0057] In some embodiments, before determining the interference of the query image set based on the first similarity matrix, the method further includes:

[0058] Retrieve target text samples from a text retrieval set; the text retrieval set contains multiple text samples.

[0059] A second similarity matrix is ​​determined based on the acquired target text samples; the second similarity matrix refers to the semantic similarity matrix between the query image set and the target text samples.

[0060] Specifically, in addition to obtaining the first similarity matrix, it is also necessary to obtain the second similarity matrix, which refers to the semantic similarity matrix between the query image set and a batch of sampled text samples.

[0061] A batch of text samples from the text retrieval set are randomly sampled as target text samples. Then, based on the similarity matrix between the image samples and the text samples or through retrieval model, the semantic similarity matrix between the image samples in the query image set and the target text samples is determined, which is the second similarity matrix.

[0062] In some embodiments, the method further includes:

[0063] The hash code of the target text sample is obtained using a retrieval model, and the hash code of the image in the query image set after perturbation is determined; the perturbation is within a preset range.

[0064] Specifically, before identifying the interference, it is necessary to use the retrieval model to obtain the hash codes of a batch of sampled text samples, i.e., the target text samples, and also to determine the hash codes of the image samples with added perturbations.

[0065] Figure 2 This is the second flowchart illustrating the online privacy protection method for retrieval models provided by this invention, as shown below. Figure 2 As shown, a batch of text samples are collected from the text retrieval set (i.e., the Database Set in the figure) as target text samples. The retrieval model generates hash codes for the target text samples and determines the hash codes corresponding to the images in the query image set (corresponding to the Query Set on the left side of the figure) with added perturbation (i.e., Adversarial Perturbation in the figure). The hash codes are stored in the form of hash tables (i.e., Hash Nets in the figure). The added (adversarial) perturbation must be within a preset range.

[0066] The generation of hash codes (i.e., Hash Code Generation in the figure) can be combined with deep convolutional networks (i.e., Deep Convolutional Networks in the figure). The deep convolutional network is used to extract image features, and a hash layer is added before the classification layer. The output of the hash layer is binarized to obtain the hash code of the image.

[0067] The online privacy protection method for retrieval models provided in this application uses semantic similarity between samples as an auxiliary supervision signal. It designs a perturbation generation method that can be used for online privacy protection, and simultaneously uses semantic similarity between image-text retrieval pairs and semantic similarity between retrieval image samples to guide the generation of interference samples. This can significantly mislead the recognition of the target model, while ensuring that the added perturbation is within a preset range, so that the perturbation will not affect the user's image recognition due to excessive size. This protects user information while ensuring a good user experience.

[0068] In some embodiments, determining the interference of the query image set based on the first similarity matrix includes:

[0069] The perturbation of the query image set is determined based on the first similarity matrix, the second similarity matrix, the hash code of the target text sample, and the hash code of the images in the query image set after perturbation.

[0070] Specifically, after determining the first similarity matrix, the second similarity matrix, the hash code of the target text sample, and the hash code of the perturbed image, the perturbation of the query image set is calculated based on these parameters.

[0071] For example, the interference of the query image set can be calculated using the following formula:

[0072]

[0073] in, Indicates interference in the image set. Indicates an iteration operation. Indicates in That is, the clipping operation under the constraint of interference range. This indicates calculating the gradient. This indicates the step size used when calculating the gradient. The constant is denoted by <>, which represents the inner product. Expressing expectations, and satisfying them. Not equal to ; Indicates the first The image sample and the first Similarity matrix between text samples An equal to 1 indicates the first The image sample and the first If two text samples have the same semantic label, a value of 0 indicates that they are not similar. It indicates that it has been added. The first disturbance Image samples The hash code; Indicates the first The hash code of a text sample. This represents the similarity matrix between image samples. An equal to 1 indicates the first The image sample and the first If two image samples have similar semantics, then the result is 0, meaning they are not similar.

[0074] In some embodiments, the method further includes:

[0075] Determine a first Hamming distance and a second Hamming distance; the first Hamming distance is the Hamming distance between the hash code of the image in the query image set after perturbation and the hash code of the first text sample, and the second Hamming distance is the Hamming distance between the hash code of the image in the query image set and the hash code of the second text sample; the first text sample and the second text sample are different text samples in the target text sample;

[0076] If the first Hamming distance is greater than the second Hamming distance, the process of re-acquiring target text samples and iteratively determining the interference of the query image set continues until the first Hamming distance is less than or equal to the second Hamming distance.

[0077] Specifically, the loss function is calculated using the Hamming distance between the hash codes of the image and the hash codes of the text (i.e., Figure 2The Retrieval Loss in the algorithm iterates over the perturbation generation process.

[0078] First, determine the Hamming distance between the hash code of the perturbated image and the hash code of the first text sample to obtain the first Hamming distance. Then, determine the Hamming distance between the hash code of the image in the query image set and the hash code of the second text sample to obtain the second Hamming distance.

[0079] The difference between the two Hamming distances is used as the loss function. If the difference between the first Hamming distance and the second Hamming distance is greater than 0, that is, the first Hamming distance is greater than the second Hamming distance, then the next round of iteration is started, a batch of text samples is resampled as new target text samples, and then the second similarity matrix and related hash codes are updated according to the new target text samples. Then, the interference of the query image set in this round is calculated, the loss is calculated, and so on.

[0080] If the difference between the first Hamming distance and the second Hamming distance is less than or equal to 0, that is, the first Hamming distance is less than or equal to the second Hamming distance, then the iteration ends, and the obtained interference is the interference finally used for image set protection.

[0081] For example, the following formula can be used to determine whether to end the iteration:

[0082]

[0083] in, This indicates the calculation of Hamming distance; It is the hash code of the l-th text sample. It is the hash code of the kth text sample that was originally paired with the i-th image sample, and l and k are not equal; Indicates the constraint range of the interference.

[0084] The online privacy protection method for retrieval models provided in this application utilizes the theoretical idea that the objective function of a hash retrieval model can be approximated through differential substitution. Semantic similarity between samples is used as an auxiliary supervision signal to design a perturbation generation method suitable for online privacy protection. This application simultaneously employs semantic similarity between image-text retrieval pairs and semantic similarity between retrieved image samples to guide the generation of interfering samples. It can select the appropriate loss function and adversarial sample generation method based on the availability of retrieval data, effectively interfering with both the hash retrieval process and the deep convolutional network's image information processing. This ensures that the generated adversarial samples can effectively generalize to other retrieval models beyond the target model. Therefore, this application can significantly mislead the target model's recognition without being detected by the human eye, and can thus be used to provide reliable image protection against powerful malicious retrieval software based on deep neural networks.

[0085] Platform administrators can use the method described in this application to mass-mask users' private images to protect the privacy of a large number of online users. Compared to traditional encoder-decoder-based privacy masking tools, hash codes eliminate the need for additional decoding processes because the added perturbations are strictly limited to a certain range. Therefore, hash codes are more convenient and suitable for large-scale social media websites. Furthermore, since invisible perturbations can be used to bypass powerful machine learning-based detectors or recognizers, hash codes provide an effective solution for attacking deep neural network (DNN)-based retrieval models and online privacy protection. Therefore, it can also be applied to other security-critical issues such as file encryption, private online conversations, or the protection of sensitive information.

[0086] Figure 3 This is a schematic diagram of the online privacy protection device for the retrieval model provided by the present invention, as shown below. Figure 3 As shown, the present invention provides an online privacy protection device for a retrieval model, including an acquisition module 301 and a determination module 302.

[0087] The acquisition module 301 is used to acquire a first similarity matrix; the first similarity matrix refers to the semantic similarity matrix between image samples in the query image set; the query image set contains multiple image samples;

[0088] The determination module 302 is used to determine the interference of the query image set based on the first similarity matrix.

[0089] Specifically, the online privacy protection device for retrieval models provided by the present invention can implement all the method steps implemented in the above-described online privacy protection method embodiments for retrieval models, and can achieve the same technical effects. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0090] It should be noted that the division of units / modules in the above embodiments of the present invention is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0091] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4As shown, the electronic device may include: a processor 401, a communications interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communications interface 402, and the memory 403 communicate with each other via the communication bus 404. The processor 401 can call logical instructions in the memory 403 to execute an online privacy protection method for a retrieval model, the method including:

[0092] Obtain the first similarity matrix; the first similarity matrix refers to the semantic similarity matrix between image samples in the query image set; the query image set contains multiple image samples;

[0093] The interference in the query image set is determined based on the first similarity matrix.

[0094] Specifically, the processor 401 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor can also adopt a multi-core architecture.

[0095] When the logical instructions in memory 403 can be implemented as software functional units and sold or used as independent products, they can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] In some embodiments, a computer program product is also provided, the computer program product including a computer program that can be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the online privacy protection method for the retrieval model provided in the above method embodiments, the method including:

[0097] Obtain the first similarity matrix; the first similarity matrix refers to the semantic similarity matrix between image samples in the query image set; the query image set contains multiple image samples;

[0098] The interference in the query image set is determined based on the first similarity matrix.

[0099] Specifically, the computer program product provided in this application embodiment can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0100] In some embodiments, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program for causing a computer to execute the online privacy protection method for a retrieval model provided in the above method embodiments, the method comprising:

[0101] Obtain the first similarity matrix; the first similarity matrix refers to the semantic similarity matrix between image samples in the query image set; the query image set contains multiple image samples;

[0102] The interference in the query image set is determined based on the first similarity matrix.

[0103] Specifically, the computer-readable storage medium provided by the present invention can implement all the method steps implemented in the above method embodiments and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0104] It should be noted that the computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic storage (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical storage (e.g., CD, DVD, BD, HVD), and semiconductor storage (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0105] It should also be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more.

[0106] In this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0107] In this invention, "determining B based on A" means that factor A must be considered when determining B. It is not limited to "B can be determined based solely on A," but should also include: "determining B based on A and C," "determining B based on A, C, and E," and "determining C based on A, and further determining B based on C," etc. Additionally, it can include using A as a condition for determining B, for example, "when A satisfies the first condition, B is determined using the first method"; or "when A satisfies the second condition, B is determined," etc.; or "when A satisfies the third condition, B is determined based on the first parameter," etc. Of course, it can also be a condition where A is a factor in determining B, for example, "when A satisfies the first condition, C is determined using the first method, and B is further determined based on C," etc.

[0108] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0109] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These processors can execute instructions that can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. An online privacy protection method for a retrieval model, characterized in that, include: Obtain the first similarity matrix; The first similarity matrix refers to the semantic similarity matrix between image samples in the query image set; The query image set contains multiple image samples; The perturbation of the query image set is determined based on the first similarity matrix; Before determining the perturbation of the query image set based on the first similarity matrix, the method further includes: Retrieve target text samples from a text retrieval set; the text retrieval set contains multiple text samples. A second similarity matrix is ​​determined based on the acquired target text samples; the second similarity matrix refers to the semantic similarity matrix between the query image set and the target text samples. The hash code of the target text sample is obtained using a retrieval model, and the hash code of the image in the query image set after perturbation is determined; the perturbation is within a preset range; The step of determining the perturbation of the query image set based on the first similarity matrix includes: The perturbation of the query image set is determined based on the first similarity matrix, the second similarity matrix, the hash code of the target text sample, and the hash code of the images in the query image set after perturbation. The method further includes: Determine a first Hamming distance and a second Hamming distance; the first Hamming distance is the Hamming distance between the hash code of the image in the query image set after perturbation and the hash code of the first text sample, and the second Hamming distance is the Hamming distance between the hash code of the image in the query image set and the hash code of the second text sample; the first text sample and the second text sample are different text samples in the target text sample; If the first Hamming distance is greater than the second Hamming distance, the process of re-acquiring the target text sample and iteratively determining the perturbation of the query image set continues until the first Hamming distance is less than or equal to the second Hamming distance.

2. The online privacy protection method for retrieval models according to claim 1, characterized in that, Obtaining the first similarity matrix includes: Based on the similarity matrix between the image sample and the text sample, extract multiple image samples that are similar to the same text sample, and obtain a first similarity matrix based on the multiple image samples that are similar to the same text sample.

3. The online privacy protection method for retrieval models according to claim 1, characterized in that, Obtaining the first similarity matrix includes: The retrieval model is used to retrieve multiple image samples corresponding to each text sample, and the first similarity matrix is ​​obtained through the multiple image samples corresponding to each text sample.

4. An online privacy protection device for a retrieval model, characterized in that, include: The acquisition module is used to obtain the first similarity matrix; The first similarity matrix refers to the semantic similarity matrix between image samples in the query image set; The query image set contains multiple image samples; The determination module is used to determine the perturbation of the query image set based on the first similarity matrix; Before determining the perturbation of the query image set based on the first similarity matrix, the determining module is further configured to: Retrieve target text samples from a text retrieval set; the text retrieval set contains multiple text samples. A second similarity matrix is ​​determined based on the acquired target text samples; the second similarity matrix refers to the semantic similarity matrix between the query image set and the target text samples. The hash code of the target text sample is obtained using a retrieval model, and the hash code of the image in the query image set after perturbation is determined; the perturbation is within a preset range; The determining module is used for: The perturbation of the query image set is determined based on the first similarity matrix, the second similarity matrix, the hash code of the target text sample, and the hash code of the images in the query image set after perturbation. The device is also used for: Determine a first Hamming distance and a second Hamming distance; the first Hamming distance is the Hamming distance between the hash code of the image in the query image set after perturbation and the hash code of the first text sample, and the second Hamming distance is the Hamming distance between the hash code of the image in the query image set and the hash code of the second text sample; the first text sample and the second text sample are different text samples in the target text sample; If the first Hamming distance is greater than the second Hamming distance, the process of re-acquiring the target text sample and iteratively determining the perturbation of the query image set continues until the first Hamming distance is less than or equal to the second Hamming distance.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the online privacy protection method for the retrieval model as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the online privacy protection method for the retrieval model as described in any one of claims 1 to 3.

Images