Cross-modal privacy semantic retrieval method, system and storage medium

Abstract

The present invention discloses a cross-modal privacy semantic retrieval method, system and storage medium, which relates to the field of data processing technology. The method includes: the client extracts semantic features from multimodal data based on a multimodal joint representation model to obtain a semantic representation vector, and encrypts the semantic representation vector to obtain a secret semantic representation vector; the server receives the secret semantic representation vector sent by the client, determines the semantic retrieval keyword associated with the secret semantic representation vector according to a preset retrieval index table, searches for the data address corresponding to the semantic retrieval keyword in the preset retrieval index table, obtains the secret semantic retrieval result, and sends the secret semantic retrieval result to the client; the client decrypts and displays the secret semantic retrieval result. The present invention solves the problem of low retrieval accuracy in the prior art, and achieves the effect of improving the accuracy of semantic retrieval results under the premise of ensuring query privacy and storage data privacy.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cross-modal privacy semantic retrieval method, system and storage medium. Background Art

[0002] With the development of Internet technology and the popularization of big data cloud service technology, in data sharing services, query and retrieval are essential operations for users to access cloud data and obtain information. However, during the query process, the user's query request and the data stored in the cloud are in plain text, and the user loses the ability to control the data. Therefore, in the data search scenario, the contradiction between big data sharing and privacy protection is more obvious.

[0003] In addition, with the advancement of Internet technology and the changes in people's production and lifestyle, the need for retrieval between multimodal data has gradually become prominent due to the diversity of data modalities. Cross-modal retrieval technology is a technology that uses a certain modality of data to search for other modal data with similar semantics by finding the semantic relationship between samples of different modalities.

[0004] In existing solutions, cross-modal retrieval can be performed on plaintext data, and corresponding modal retrieval can be performed on private data, but there are few solutions for cross-modal retrieval of private data. The method of cross-modal semantic retrieval of private data has the problem that the extracted keywords have no semantic association with the search content, and the search results may be biased, resulting in low accuracy of cross-modal privacy semantic retrieval. Summary of the invention

[0005] The main purpose of the present invention is to provide a cross-modal privacy semantic retrieval method, system and storage medium, aiming to solve the technical problem that the existing methods for retrieving multimodal privacy data have low accuracy.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a cross-modal privacy semantic retrieval method, which is applied to a server, and the method comprises:

[0008] Receiving a secret semantic representation vector sent by a client; wherein the secret semantic representation vector is obtained by extracting semantic features from multimodal data based on a multimodal joint representation model by the client, and encrypting the semantic representation vector;

[0009] Determine the semantic search keywords associated with the dense semantic representation vector according to a preset search index table; wherein the preset search index table includes a mapping relationship between the semantic search keywords and the data addresses of the stored data;

[0010] The data address corresponding to the semantic search keyword is searched in the preset search index table to obtain the encrypted semantic search result, and the encrypted semantic search result is sent to the client so that the client can decrypt and display the encrypted semantic search result.

[0011] Optionally, in the above cross-modal privacy semantic retrieval method, the step of determining the semantic retrieval keyword associated with the secret semantic representation vector according to a preset retrieval index table includes:

[0012] According to a preset search index table, determining the association value between the dense semantic representation vector and all search keywords in the preset search index table;

[0013] According to the association value, semantic search keywords associated with the dense semantic representation vector are determined; wherein the semantic search keywords include search keywords sorted according to the association value.

[0014] Optionally, in the above cross-modal privacy semantic retrieval method, before the step of receiving the secret semantic representation vector sent by the client, the method further includes:

[0015] Receiving the secret data to be stored and the corresponding secret keywords sent by the client; wherein the secret data to be stored is obtained by encrypting the data to be stored by the client, and the secret keywords are obtained by extracting semantic features of the data to be stored by the client based on a multimodal joint representation model, and encrypting the keywords;

[0016] Storing the secret data to be stored, and obtaining the stored data and its data address;

[0017] An index table is constructed according to the secret keywords and the data addresses of the stored data to obtain a preset retrieval index table.

[0018] Optionally, in the above cross-modal privacy semantic retrieval method, the step of constructing an index table according to the secret keyword and the data address of the stored data to obtain a preset retrieval index table includes:

[0019] Constructing an index table according to the secret key words and the data addresses of the stored data to obtain a retrieval index table;

[0020] Determining the similarity values ​​between the secret keywords according to the search index table;

[0021] According to the similarity value, clustering processing is performed to obtain semantic search keywords; wherein the semantic search keywords include a cluster of dense keywords whose similarity value is less than a preset threshold;

[0022] A preset search index table is obtained according to the mapping relationship between the semantic search keywords and the corresponding data addresses of the stored data.

[0023] In a second aspect, the present invention provides a cross-modal privacy semantic retrieval method, which is applied to a client, and the method includes:

[0024] Acquire multimodal data;

[0025] Extracting semantic features from the multimodal data using a multimodal joint representation model to obtain a semantic representation vector;

[0026] Encrypting the semantic representation vector to obtain a encrypted semantic representation vector;

[0027] The dense semantic representation vector is sent to the server, so that the server determines the semantic search keyword associated with the dense semantic representation vector according to a preset search index table, searches the data address corresponding to the semantic search keyword in the preset search index table, obtains the dense semantic search result, and sends the dense semantic search result to the client; wherein the preset search index table includes a mapping relationship between the semantic search keyword and the data address of the stored data;

[0028] Receive the encrypted semantic retrieval result sent by the server, and decrypt and display the encrypted semantic retrieval result.

[0029] Optionally, in the above cross-modal privacy semantic retrieval method, the step of extracting semantic features from the multimodal data through a multimodal joint representation model to obtain a semantic representation vector includes:

[0030] Preprocessing the multimodal data to obtain preprocessed multimodal data; wherein the preprocessing includes any one of format switching, text encoding, size scaling and noise elimination;

[0031] The preprocessed multimodal data is input into a multimodal joint representation model to output a semantic feature vector; wherein the multimodal joint representation model includes a representation learning model for performing modality conversion on the multimodal data.

[0032] In a third aspect, the present invention provides a server, comprising a processor and a memory, wherein a cross-modal privacy semantic retrieval program is stored in the memory, and when the cross-modal privacy semantic retrieval program is executed by the processor, the cross-modal privacy semantic retrieval method as described above is implemented.

[0033] In a fourth aspect, the present invention provides a client, comprising a processor and a memory, wherein a cross-modal privacy semantic retrieval program is stored in the memory, and when the cross-modal privacy semantic retrieval program is executed by the processor, the cross-modal privacy semantic retrieval method as described above is implemented.

[0034] In a fifth aspect, the present invention provides a cross-modal privacy semantic retrieval system, the system comprising:

[0035] Such as the server mentioned above;

[0036] As the client mentioned above;

[0037] The server is in communication connection with the client.

[0038] In a sixth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and the computer program can be executed by one or more processors to implement the cross-modal privacy semantic retrieval method as described above.

[0039] The above one or more technical solutions provided by the present invention may have the following advantages or at least achieve the following technical effects:

[0040] The present invention proposes a cross-modal privacy semantic retrieval method, system and storage medium. The client extracts semantic features from multimodal data based on a multimodal joint representation model to obtain a semantic representation vector, then encrypts the semantic representation vector to obtain a secret semantic representation vector, and sends it to the server. The server determines the semantic retrieval keyword associated with the secret semantic representation vector according to a preset retrieval index table, and then searches the preset retrieval index table for the data address corresponding to the semantic retrieval keyword to obtain the secret semantic retrieval result, which is returned to the client. The client decrypts and displays the secret semantic retrieval result to achieve the purpose of privacy semantic retrieval of multimodal data. The present invention does not need to rely on a keyword library for semantic representation, and can ensure the semantic relevance between the retrieval keyword and the retrieval result, thereby achieving the effect of improving the accuracy of the semantic retrieval result while ensuring the privacy of the query and the privacy of the stored data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 This is a flow chart of the first embodiment of the cross-modal privacy semantic retrieval method of the present invention;

[0043] Figure 2 A schematic diagram of the hardware structure of a server involved in the present invention;

[0044] Figure 3 This is a flow chart of the second embodiment of the cross-modal privacy semantic retrieval method of the present invention;

[0045] Figure 4 A schematic diagram of the hardware structure of the client involved in the present invention;

[0046] Figure 5 Schematic diagram of the functional modules of the cross-modal privacy semantic retrieval system of the present invention.

[0047] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that in the present invention, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the existence of other identical elements in the process, method, article or system including the element. In addition, in the present invention, the use of suffixes such as "module", "component" or "unit" used to represent elements is only to facilitate the description of the present invention, and has no specific meaning in itself. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0050] For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0051] Analysis of existing technologies shows that with the development of Internet technology and the popularization of big data cloud service technology, information resources and services face the risk of data security and privacy leakage in the process of data sharing, resulting in a contradiction between data fusion and security and privacy. Especially in data search scenarios, the contradiction between big data sharing and privacy protection is particularly obvious.

[0052] Query retrieval is an essential operation for users to access cloud data and obtain information. However, during the query process, since big data is in the cloud, the query request needs to be uploaded to the cloud and retrieved by the cloud service provider. During this process, the user's query request and the data stored in the cloud are in plain text, and the user loses the ability to control the data. Therefore, in the current untrusted environment of the Internet, the user's query request, cloud data and request results are always facing threats such as tampering, deletion, and theft, which can easily cause privacy leakage, data tampering and other privacy security issues.

[0053] In addition, with the advancement of Internet technology and the changes in people's production and lifestyle, Internet application scenarios have become richer, various applications have penetrated into every corner of people's lives, and data modalities have become more diverse. Due to the diversity of data modalities, the retrieval needs between multimodal data have gradually become prominent. Cross-modal retrieval technology is to use a certain modal sample to search for samples of other modalities with similar semantics by finding the semantic relationship between samples of different modalities. For example, a query request of one modality (such as text) is used to retrieve data content with similar semantics but in other modalities (such as images). Data modalities are very diverse. In addition to common text, images, voice, etc., videos, physiological signals, point cloud data, etc. can all be used as part of the retrieval content in cross-modal retrieval.

[0054] In the existing technical solutions, cross-modal retrieval can be performed on plaintext data, and corresponding modal retrieval can also be performed on private data, but there are few solutions for cross-modal retrieval of private data. There are still some problems with the method of cross-modal retrieval of private data:

[0055] 1. The structure of multimodal data is complex, making it difficult to effectively extract keywords;

[0056] 2. The keyword extraction of privacy data relies on a predefined keyword library. The semantics of the extracted keywords are independent of each other and have poor relevance.

[0057] 3. When searching based on keywords subsequently, only the keyword is searched, and other words associated with the keyword are not searched, resulting in no semantic association between the keyword and the search content, possible deviations in the search results, and low accuracy of cross-modal retrieval.

[0058] In view of the technical problem that the methods for retrieving multimodal private data in the prior art have low accuracy, the present invention provides a cross-modal private semantic retrieval method, and the overall idea is as follows:

[0059] The client obtains multimodal data; extracts semantic features from the multimodal data through a multimodal joint representation model to obtain a semantic representation vector; encrypts the semantic representation vector to obtain a secret semantic representation vector; sends the secret semantic representation vector to a server; receives a secret semantic retrieval result sent by the server, and decrypts and displays the secret semantic retrieval result.

[0060] The server receives a dense semantic representation vector sent by a client; determines a semantic search keyword associated with the dense semantic representation vector according to a preset retrieval index table; wherein the preset retrieval index table includes a mapping relationship between semantic search keywords and data addresses of stored data; searches for the data address corresponding to the semantic search keyword in the preset retrieval index table, obtains a dense semantic retrieval result, and sends the dense semantic retrieval result to the client.

[0061] Through the above technical solution, the client first extracts semantic features from multimodal data based on a multimodal joint representation model to obtain a semantic representation vector, then encrypts the semantic representation vector to obtain a secret semantic representation vector, and sends it to the server. The server determines the semantic search keyword associated with the secret semantic representation vector according to a preset search index table, and then searches the preset search index table for the data address corresponding to the semantic search keyword to obtain a secret semantic search result, which is returned to the client. The client decrypts and displays the secret semantic search result to achieve the purpose of privacy semantic retrieval of multimodal data. The present invention does not need to rely on a keyword library for semantic representation, and can ensure the semantic relevance between the search keyword and the search result, thereby achieving the effect of improving the accuracy of the semantic search result while ensuring the privacy of the query and the privacy of the stored data.

[0062] The cross-modal privacy semantic retrieval method, system and storage medium provided by the present invention are described in detail below through specific embodiments and implementation modes in conjunction with the accompanying drawings.

[0063] Embodiment 1

[0064] Reference Figure 1 The flowchart of the present invention proposes a first embodiment of the cross-modal privacy semantic retrieval method of the present invention, which is applied to a server. The server can be a network device capable of achieving network connection, or a cloud service platform capable of achieving network connection.

[0065] like Figure 2The server may include: a processor 1001, such as a CPU (Central Processing Unit), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Those skilled in the art will appreciate that Figure 2 The hardware structure shown in the figure does not constitute a limitation on the server of the present invention, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0066] Specifically, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 is used to connect to the user terminal and communicate data with the user terminal. The user interface 1003 may include an output unit, such as a display screen, an input unit, such as a keyboard, and optionally, the user interface 1003 may also include other input / output interfaces, such as a standard wired interface and a wireless interface; the network interface 1004 is used to connect to the background server and communicate data with the background server. The network interface 1004 may include an input / output interface, such as a standard wired interface and a wireless interface, such as a Wi-Fi interface; the memory 1005 is used to store various types of data, which may include, for example, instructions of any application or method in the server, and data related to the application. The memory 1005 may be a high-speed RAM memory, or a stable memory, such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the processor 1001;

[0067] For details, please refer to Figure 2 The memory 1005 may include an operating system, a network communication module, a user interface module, and a cross-modal privacy semantic retrieval program, wherein the network communication module is mainly used to connect to the client and perform data communication with the client; the processor 1001 is used to call the cross-modal privacy semantic retrieval program stored in the memory 1005, and perform the following operations:

[0068] Receiving a secret semantic representation vector sent by a client; wherein the secret semantic representation vector is obtained by extracting semantic features from multimodal data based on a multimodal joint representation model by the client, and encrypting the semantic representation vector;

[0069] Determine the semantic search keywords associated with the dense semantic representation vector according to a preset search index table; wherein the preset search index table includes a mapping relationship between the semantic search keywords and the data addresses of the stored data;

[0070] The data address corresponding to the semantic search keyword is searched in the preset search index table to obtain the encrypted semantic search result, and the encrypted semantic search result is sent to the client so that the client can decrypt and display the encrypted semantic search result.

[0071] Based on the above server, the following is combined Figure 1 The flowchart shown in the figure describes in detail the cross-modal privacy semantic retrieval method of this embodiment. The method may include the following steps:

[0072] Step S120: receiving a secret semantic representation vector sent by the client; wherein the secret semantic representation vector is obtained by extracting semantic features from multimodal data based on a multimodal joint representation model by the client, and encrypting the semantic representation vector.

[0073] Specifically, the server and the client are connected in communication to form a retrieval system. In the retrieval system, the user can directly input the data to be retrieved on the client, and the input data can be multimodal data, that is, data information including multiple different modalities. Multimodal data can include one or more of different modal data such as text data, image data, voice data, video data, and physiological signal data, and can be obtained through different applications on the client.

[0074] After the client obtains the multimodal data, it can preprocess the multimodal data, including but not limited to data regularization, format switching, noise elimination, encoding and decoding, size scaling, etc., and preprocess the preprocessed data on the basis of being able to directly input the preprocessed data into the multimodal joint representation model. The specific processing content can be set according to the actual situation. In this embodiment, multimodal data including text data and image data is taken as an example for explanation.

[0075] Assuming that the text data is a long sequence consisting of English letters, numbers, and symbols, when the client's multimodal joint representation model uses the trained BERT (Bidirectional Encoder Representation from Transformers) model to encode the text data, the BPE (Byte Pair Encoder) method can be used to preprocess the text data in the multimodal data, that is, pre-encode it, and then input the pre-encoded text data into the BERT model.

[0076] The BPE method is an algorithm that encodes based on byte pairs, which can compress text data. The BPE method is an iterative process in which a pair of characters with the highest frequency in a string is replaced by a character that does not appear in the string. Taking English as an example, the algorithm will split the training corpus into characters, combine them according to character pairs, and sort the results of all combinations according to the frequency of occurrence. The higher the frequency of occurrence, the higher the ranking. The first character is the character pair with the highest frequency of occurrence, and then another character is selected to replace the character pair to achieve the purpose of compressing the text data. In this embodiment, the text data is preprocessed by this method to obtain preprocessed text data. The client then inputs the preprocessed text data into the BERT model, segmentes the text, obtains a subword sequence, and then obtains a subword index sequence based on the subwords and subword indexes in the dictionary of the BERT model to achieve the encoding of the text data, that is, to achieve feature extraction of the text data, and obtain the corresponding semantic representation vector, which is a 768-dimensional row vector.

[0077] In order to solve the problem of semantic retrieval under privacy protection of cross-modal data and ensure the privacy security of user data, the client also encrypts the extracted semantic representation vector. Specifically, it uses an encryption method that supports multiple correlation calculations to obtain the encrypted semantic representation vector corresponding to the text data.

[0078] Similarly, the image data in the multimodal data is also preprocessed and feature extracted in turn. When the client's multimodal joint representation model uses the ResNet50 (Residual Network) model to encode the image data, since the model has requirements on the size of the input data, the transform method of the PyTorch framework in the open source Python machine learning library can be used to preprocess the image data in the multimodal data, that is, to scale the size, and then input the scaled image data into the ResNet50 model. In this embodiment, after the image data is preprocessed by this method, the preprocessed image data is obtained. The client then inputs the preprocessed image data into the ResNet50 model to realize the encoding of the image data, that is, to realize the feature extraction of the image data, and obtain the corresponding semantic representation vector, which is a 768-dimensional row vector. The client also continues to encrypt the semantic representation vector to obtain a dense semantic representation vector of the image data.

[0079] According to the above method, after the client preprocesses the multimodal data, the preprocessed multimodal data is obtained, and then the preprocessed multimodal data is input into the multimodal joint representation model to encode the multimodal data. After obtaining the dense semantic representation vector corresponding to the multimodal data, the dense semantic representation vector is sent to the server, and the server correspondingly receives the dense semantic representation vector sent by the client.

[0080] Step S140: Determine the semantic search keywords associated with the dense semantic representation vector according to a preset search index table; wherein the preset search index table includes a mapping relationship between the semantic search keywords and the data addresses of the stored data.

[0081] In the preset retrieval index table, there may be multiple data addresses of stored data that have a mapping relationship with the semantic retrieval keywords, or it may include a cluster of retrieval keywords and their corresponding stored data addresses. That is to say, in the preset retrieval index table, there may be a one-to-many relationship between the semantic retrieval keywords and the data addresses, or there may be a one-to-many relationship between the semantic retrieval keywords and the retrieval keywords, and the retrieval keywords may have a one-to-many relationship with the data addresses. When constructing the preset retrieval index table of this structure, a clustering algorithm may be used for clustering processing. Compared with the method of comparing with each retrieval keyword separately, the clustered preset retrieval index table can prevent the index table size from growing linearly with the data size, thereby reducing the retrieval complexity during retrieval.

[0082] Specifically, step S140 may include:

[0083] Step S141: According to a preset search index table, determining the association value between the dense semantic representation vector and all search keywords in the preset search index table.

[0084] The calculation formula of the correlation value is:

[0085]

[0086] Among them, En() represents the encryption method, En(e) represents the secret semantic representation vector, En(e′) represents the search keyword in the preset search index table. It should be noted that the search keyword is secret, n represents the number, and i represents any one of the number n.

[0087] The server calculates the association value between the dense semantic representation vector and all the search keywords in the preset search index table according to the preset search index table and the above-mentioned association value calculation formula.

[0088] Step S142: Determine semantic search keywords associated with the dense semantic representation vector according to the association value; wherein the semantic search keywords include search keywords sorted according to the association value.

[0089] After calculating the association value between the dense semantic representation vector and all search keywords in the preset search index table, the server sorts them in descending order according to the association value. The sorting result is the semantic search keyword associated with the dense semantic representation vector.

[0090] Step S160: Searching for a data address corresponding to the semantic search keyword in the preset search index table, obtaining a secret semantic search result, and sending the secret semantic search result to the client, so that the client decrypts and displays the secret semantic search result.

[0091] The server sorts the semantic retrieval keywords according to the calculated association values, and searches the preset retrieval index table for the data address corresponding to each semantic retrieval keyword, thereby obtaining the confidential semantic retrieval results in the form of a list. The server sends the confidential semantic retrieval results to the client. After receiving the retrieval results, the client uses the same encryption method as the encryption operation on the extracted semantic representation vector to decrypt the retrieval results, and then displays them on the client. During the entire cross-modal privacy semantic retrieval process, all operations performed on the server are performed in an encrypted state, which can ensure the security of data privacy. The retrieval results returned to the client are decrypted and displayed to the user in plain text, so that the user can intuitively know the retrieval results.

[0092] In one embodiment, before step S120, the method further includes:

[0093] Step S111: receiving the secret data to be stored and the corresponding secret keywords sent by the client; wherein the secret data to be stored is obtained by encrypting the data to be stored by the client, and the secret keywords are obtained by extracting semantic features of the data to be stored by the client based on a multimodal joint representation model, and encrypting the keywords;

[0094] Before the above-mentioned retrieval system performs actual retrieval operations, the server needs to use a preset retrieval index table and stored data. Therefore, before step S120, it is necessary to first establish a preset retrieval index table and store user privacy data on the server, that is, to store the confidential storage data uploaded by the user to the server through the client.

[0095] The server receives the confidential data to be stored sent by the client, and the confidential data to be stored is uploaded directly to the server by the client after encrypting the data to be stored, wherein the data to be stored can be unimodal data or multimodal data; the server receives the confidential keywords corresponding to the confidential data to be stored sent by the client, and the confidential keywords are the semantic feature vectors obtained by the client after preprocessing, keyword extraction and encryption of the data to be stored. Among them, the specific steps of the client extracting and encrypting the semantic features of the data to be stored through the multimodal joint representation model can be referred to the specific steps of the client extracting and encrypting the semantic features of the multimodal data in step S120, which will not be repeated here. It should be noted that, unlike the aforementioned steps, it is necessary to encrypt the data to be stored separately here to obtain the confidential data to be stored, which is sent to the server for storage as a database for subsequent retrieval.

[0096] In this embodiment, the client uses a multimodal joint representation model to replace the traditional keyword extraction model with text as input, and can obtain a representation vector containing rich semantic features. The representation vector is used as the extracted keyword, overcoming the defect of the traditional method that the retrieval keyword and the retrieval result have no semantic association; at the same time, a secure KNN searchable encryption method is used to encrypt the extracted keywords, so that the encryption method supports multiple correlation calculations.

[0097] Step S112: store the secret data to be stored, and obtain the stored data and its data address.

[0098] After receiving the secret data to be stored, the server stores the secret data to be stored in the database, and the data becomes the stored data. Combined with the storage address of the data in the database, the server obtains the stored data and its data address.

[0099] Step S113: construct an index table according to the secret keyword and the data address of the stored data to obtain a preset retrieval index table.

[0100] Due to the modal complexity of the stored data, when searching and constructing the search index table, the stored data is not directly used as the search object, but the storage address of the stored data is used as the search object associated with the secret key words. Therefore, a preset search index table including the mapping relationship between the semantic search key words and the data addresses of the stored data can be obtained.

[0101] The client uses a multimodal joint representation model to extract semantic representation vectors for the data to be stored, and then uses a data encryption method to encrypt the extracted semantic representation vectors and the data to be stored respectively. After the encrypted semantic representation vectors, i.e., the secret keywords, and the encrypted data to be stored, i.e., the secret data to be stored, are uploaded to the server, the server can construct a retrieval index table based on the secret keywords and the data addresses of the secret data to be stored to obtain a preset retrieval index table.

[0102] The user can also store other data on the server according to steps S111 to S113 to enrich the stored data in the database, that is, to enrich the searched objects; and update the retrieval index table on the server so that the updated preset retrieval index table can be used directly during the next retrieval, so that the retrieval results are real-time and accurate.

[0103] Specifically, step S113 may include:

[0104] Step S113.1: construct an index table according to the secret keyword and the data address of the stored data to obtain a retrieval index table;

[0105] The search index table includes a one-to-one or one-to-many mapping relationship between secret keywords and data addresses.

[0106] Step S113.2: Determine the similarity values ​​between the secret keywords according to the search index table;

[0107] Step S113.3: performing clustering processing according to the similarity value to obtain semantic search keywords; wherein the semantic search keywords include a cluster of dense keywords whose similarity value is less than a preset threshold;

[0108] Step S113.4: Obtain a preset search index table according to the mapping relationship between the semantic search keywords and the corresponding data addresses of the stored data.

[0109] Specifically, directly establishing a mapping relationship between the stored data and its corresponding secret keywords, a one-to-one corresponding index table may have a lot of content and a complex structure, which is not convenient for the subsequent rapid semantic retrieval. Therefore, the index table can be clustered. First, calculate the similarity values ​​between the secret keywords in the retrieval index table constructed in step S113.1, and identify the secret keywords that meet the preset requirements, such as meeting the preset threshold, as a class or a cluster with semantic relevance, and then summarize these secret keywords to obtain an index sub-table, and then set an index corresponding to the index sub-table, that is, to indicate that this class or this cluster of secret keywords is a semantic retrieval keyword, thereby obtaining a preset retrieval index table for use in steps S140 and S160.

[0110] Table 1 shows an example of a preset search index table obtained after clustering according to the above method:

[0111] Table 1

[0112]

[0113] Based on the preset retrieval index table, when searching, it is only necessary to calculate the association value of the dense semantic representation vector with the semantic retrieval keyword 1, semantic retrieval keyword 2, ..., semantic retrieval keyword m respectively, so as to obtain the dense semantic retrieval result after sorting the semantic retrieval keyword 1, semantic retrieval keyword 2, ..., semantic retrieval keyword m.

[0114] It should be noted that the above steps S111 to S113 may be independently executed step branches, thereby achieving the purpose of data uploading; or steps S120 to S160 may be executed in sequence after steps S111 to S113, thereby achieving the purpose of data retrieval.

[0115] It should be noted that the method used for data encryption and decryption on the client in this method can be a conventional symmetric encryption method. The requirement for selecting the encryption method is to ensure that the cosine similarity cos(e, e′) between the keyword e obtained by the client during the data upload process to extract the semantic features of the data to be stored and the semantic representation vector e′ obtained by the client during the data retrieval process to extract the semantic features of the multimodal data is consistent with the cosine similarity cos(En(e), En(e′)) between the secret keyword En(e) obtained by the client during the data upload process and the secret semantic representation vector En(e′) obtained by the client during the data retrieval process, that is, to satisfy Formula 1:

[0116] cos(e,e′)=cos(En(e),En(e′)),

[0117] Among them, the cosine similarity calculation formula is:

[0118]

[0119] Therefore, Formula 1 can be transformed into Formula 2:

[0120]

[0121] Keyword e and semantic representation vector e′, dense keyword En(e) and dense semantic representation vector En(e′) are both vectors. After L2 norm normalization of the vector, the Euclidean distance of the vector is 1, that is,

[0122] Therefore, Formula 2 can be rewritten as Formula 3:

[0123]

[0124] It can be concluded that after L2 norm normalization of the vector, the cosine similarity cos(e,e′) between the keyword e and the semantic representation vector e′ is equal to the direct multiplication of the two vectors. Formula 3 can be further converted to Formula 4:

[0125] cos(e,e′)=cos(En(e),En(e′))=e·e′=En(e)·En(e′),

[0126] At this point, we only need to solve the encryption method to ensure that Formula 6 is valid. According to the theorem that the multiplication of two mutually inverse matrices is 1, the matrix M can be used as the encryption matrix corresponding to the encryption method, and the inverse transpose of the matrix M -1T As a decryption matrix, there is formula 5:

[0127] e·e′=En(e)·En(e′)=(e·M)·(e′·M -1T ),

[0128] Among them, the vector e is multiplied by the encryption matrix M to obtain the encryption vector En(e), that is, the secret keyword, and the vector e′ is multiplied by the decryption matrix M -1T , we get the encrypted vector En(e′), which is the secret semantic representation vector; the multiplication of plaintext vectors is equivalent to the multiplication of ciphertext vectors. This shows that the client can use the randomly generated encryption matrix M as the encryption key, and the inverse transposed matrix M -1T As a decryption matrix, it is used for encryption and decryption operations on the client side.

[0129] The method of this embodiment can effectively improve the accuracy and semantic relevance of retrieval results while ensuring query privacy and cloud data privacy security.

[0130] The cross-modal privacy semantic retrieval method provided in this embodiment is to extract semantic features of multimodal data based on a multimodal joint representation model by the client to obtain a semantic representation vector, and then encrypt the semantic representation vector to obtain a secret semantic representation vector, and send it to the server. The server determines the semantic retrieval keyword associated with the secret semantic representation vector according to a preset retrieval index table, and then searches the data address corresponding to the semantic retrieval keyword in the preset retrieval index table to obtain the secret semantic retrieval result, and returns it to the client. The client decrypts and displays the secret semantic retrieval result to achieve the purpose of privacy semantic retrieval of multimodal data. The present invention does not need to rely on a keyword library for semantic representation, and can ensure the semantic relevance between the retrieval keyword and the retrieval result, thereby achieving the effect of improving the accuracy of the semantic retrieval result under the premise of ensuring the privacy of the query and the privacy of the stored data.

[0131] Embodiment 2

[0132] Based on the same inventive concept, Figure 1 The flowchart of the present invention proposes a second embodiment of the cross-modal privacy semantic retrieval method of the present invention, which is applied to a client. The client refers to a terminal device that can achieve network connection, and the client can be a mobile phone, computer, tablet computer, portable computer and other terminal devices.

[0133] like Figure 4 , which is a schematic diagram of the hardware structure of the client. The client may include: a processor 2001, such as a CPU (Central Processing Unit), a communication bus 2002, a user interface 2003, a network interface 2004, and a memory 2005. Those skilled in the art will appreciate that Figure 4 The hardware structure shown in the figure does not constitute a limitation on the client of the present invention, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0134] Specifically, the communication bus 2002 is used to realize the connection and communication between these components; the user interface 2003 is used to connect to other user terminals and communicate data with other user terminals. The user interface 2003 may include an output unit, such as a display screen, an input unit, such as a keyboard, and optionally, the user interface 2003 may also include other input / output interfaces, such as a standard wired interface and a wireless interface; the network interface 2004 is used to connect to the background server and communicate data with the background server. The network interface 2004 may include an input / output interface, such as a standard wired interface and a wireless interface, such as a Wi-Fi interface; the memory 2005 is used to store various types of data, which may include, for example, instructions of any application or method in the server, and data related to the application. The memory 2005 may be a high-speed RAM memory, or a stable memory, such as a disk memory. Optionally, the memory 2005 may also be a storage device independent of the processor 2001;

[0135] For details, please refer to Figure 4 The memory 2005 may include an operating system, a network communication module, a user interface module, and a cross-modal privacy semantic retrieval program, wherein the network communication module is mainly used to connect to a server and perform data communication with the server; the processor 1001 is used to call the cross-modal privacy semantic retrieval program stored in the memory 2005, and perform the following operations:

[0136] Based on the above client, the following is combined Figure 3The flowchart shown in the figure describes in detail the cross-modal privacy semantic retrieval method of this embodiment. The method may include the following steps:

[0137] Step S210: Acquire multimodal data.

[0138] The user can directly input the data that he wants to retrieve on the client. The input data can be multimodal data, that is, data information including multiple different modalities. Multimodal data can include one or more of different modal data such as text data, image data, voice data, video data, and physiological signal data, which can be obtained through different applications on the client.

[0139] Step S230: extracting semantic features from the multimodal data using a multimodal joint representation model to obtain a semantic representation vector.

[0140] Specifically, step S230 may include:

[0141] Step S231: preprocessing the multimodal data to obtain preprocessed multimodal data; wherein the preprocessing includes any one of format switching, text encoding, size scaling and noise elimination.

[0142] Preprocessing can organize multimodal data into data that can be directly input into the multimodal joint representation model to prevent the model from being unable to extract feature vectors due to the data input into the model not meeting the requirements. The specific content of preprocessing can be set according to the actual situation of the modal types in the multimodal data. For example, when the multimodal data includes text data and image data, the BPE method can be used to preprocess the text data to obtain preprocessed text data, and the transform method can be used to preprocess the image data to obtain preprocessed image data, and then the preprocessed different modal data can be aggregated to obtain preprocessed multimodal data.

[0143] Step S232: input the preprocessed multimodal data into a multimodal joint representation model, and output a semantic feature vector; wherein the multimodal joint representation model includes a representation learning model for performing modality conversion on the multimodal data.

[0144] Specifically, the multimodal joint representation model may include a BERT model that performs feature vectorization encoding on text data, or a ResNet50 model that performs feature vectorization encoding on image data. Specifically, models may be selected separately according to the types of modalities in the multimodal data, and then these models may be aggregated into a multimodal joint representation model to achieve feature vector extraction of multimodal data.

[0145] The client inputs the preprocessed multimodal data into the multimodal joint representation model, encodes the multimodal data, and obtains the semantic feature vector of the multimodal data.

[0146] Step S250: encrypt the semantic representation vector to obtain a encrypted semantic representation vector.

[0147] In order to ensure data privacy and security during cross-modal data retrieval, the semantic representation vector needs to be encrypted before the client uploads it to the server. The encryption method needs to support multiple correlation calculation methods to ensure that the correlation between the semantic representation vectors before encryption and the encrypted semantic representation vectors of different modal data is consistent. This can prevent the encryption operation from causing changes in the semantic representation vector and affecting the accuracy of subsequent retrieval.

[0148] Step S270: Send the dense semantic representation vector to the server, so that the server determines the semantic search keyword associated with the dense semantic representation vector according to a preset search index table, searches the data address corresponding to the semantic search keyword in the preset search index table, obtains the dense semantic search result, and sends the dense semantic search result to the client; wherein the preset search index table includes a mapping relationship between the semantic search keyword and the data address of the stored data;

[0149] Step S290: receiving the encrypted semantic retrieval result sent by the server, and decrypting and displaying the encrypted semantic retrieval result.

[0150] In one embodiment, the method may further include:

[0151] Step S221: determining the multimodal data as data to be stored;

[0152] When a user needs to upload data, he only needs to perform corresponding operations on the client to import the multimodal data, and the multimodal data can be determined as data to be stored, and will not be considered as multimodal data to be retrieved;

[0153] Step S222: encrypting the data to be stored to obtain encrypted data to be stored;

[0154] The client encrypts the data to be stored and obtains the confidential data to be stored;

[0155] Step S223: extracting semantic features of the data to be stored based on the multimodal joint representation model to obtain keywords;

[0156] Specifically, it may also include preprocessing the data to be stored and then inputting it into the multimodal joint representation model to extract semantic features, and determining the extracted feature vectors as keywords corresponding to the data to be stored;

[0157] Step S224: encrypt the keyword to obtain a secret keyword;

[0158] Step S225: Send the secret data to be stored and the secret keywords to the server, so that after receiving the secret data to be stored and the corresponding secret keywords, the server stores the secret data to be stored, obtains the stored data and its data address, and constructs an index table according to the secret keywords and the data address of the stored data to obtain a preset retrieval index table.

[0159] For more implementation details in the specific implementation of the above method steps, please refer to the description of the specific implementation in Example 1. For the sake of brevity of the description, they will not be repeated here.

[0160] The cross-modal privacy semantic retrieval method provided in this embodiment does not need to rely on a predefined keyword library when performing keyword extraction on the client. It can directly use the semantic keywords of the stored data uploaded by the user to the server through the client as the index in the retrieval index table, and directly calculate the correlation with the secret semantic representation vector, so as to retrieve multi-modal heterogeneous data with semantic feature privacy with the best performance. Under the premise of ensuring the security of query privacy and cloud data privacy, the accuracy of the retrieval results can meet the requirements of cross-modal data retrieval that ensures user privacy security.

[0161] Embodiment 3

[0162] Based on the same inventive concept, Figure 2 , which is a schematic diagram of the hardware structure of the server of the present invention. This embodiment provides a server, which may include a processor and a memory, wherein a cross-modal privacy semantic retrieval program is stored in the memory, and when the cross-modal privacy semantic retrieval program is executed by the processor, all or part of the steps of the first embodiment of the cross-modal privacy semantic retrieval method of the present invention are implemented.

[0163] Specifically, the server may be a network device capable of achieving network connection, or may be a cloud service platform capable of achieving network connection.

[0164] It can be understood that the server may also include a communication bus, a user interface and a network interface. Among them, the communication bus is used to realize the connection and communication between these components. The user interface is used to connect to the user terminal and communicate data with the user terminal. The user interface may include an output unit, such as a display screen, and an input unit, such as a keyboard. The network interface is used to connect to the backend server and communicate data with the backend server. The network interface may include an input / output interface, such as a standard wired interface, a wireless interface, such as a Wi-Fi interface.

[0165] The memory is used to store various types of data, which may include, for example, instructions of any application or method in the server, and data related to the application. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. Optionally, the memory may also be a storage device independent of the processor.

[0166] The processor is used to call the cross-modal privacy semantic retrieval program stored in the memory and execute the cross-modal privacy semantic retrieval method. The processor can be a digital signal processor (Digital Signal Processor, referred to as DSP), a programmable logic device (Programmable Logic Device, referred to as PLD), a field programmable gate array (Field Programmable Gate Array, referred to as FPGA), a controller, a microcontroller, a microprocessor or other electronic components, used to execute all or part of the steps of the first embodiment of the cross-modal privacy semantic retrieval method mentioned above.

[0167] Embodiment 4

[0168] Based on the same inventive concept, Figure 4, which is a hardware structure diagram of the client of the present invention. This embodiment provides a client, which may include a processor and a memory, wherein a cross-modal privacy semantic retrieval program is stored in the memory, and when the cross-modal privacy semantic retrieval program is executed by the processor, all or part of the steps of the second embodiment of the cross-modal privacy semantic retrieval method of the present invention are implemented.

[0169] Specifically, the client refers to a terminal device that can achieve network connection, which can be a mobile phone, computer, tablet computer, portable computer and other terminal devices.

[0170] It can be understood that the client can also include a communication bus, a user interface and a network interface. Among them, the communication bus is used to realize the connection and communication between these components. The user interface is used to connect to other client terminals and communicate data with other client terminals. The user interface can include an output unit, such as a display screen, and an input unit, such as a keyboard. The network interface is used to connect to the backend server and communicate data with the backend server. The network interface can include an input / output interface, such as a standard wired interface, a wireless interface, such as a Wi-Fi interface.

[0171] The memory is used to store various types of data, which may include, for example, instructions of any application or method in the client, and data related to the application. The memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. Optionally, the memory may also be a storage device independent of the processor.

[0172] The processor is used to call the cross-modal privacy semantic retrieval program stored in the memory and execute the cross-modal privacy semantic retrieval method. The processor can be a digital signal processor (Digital Signal Processor, referred to as DSP), a programmable logic device (Programmable Logic Device, referred to as PLD), a field programmable gate array (Field Programmable Gate Array, referred to as FPGA), a controller, a microcontroller, a microprocessor or other electronic components, used to execute all or part of the steps of the second embodiment of the cross-modal privacy semantic retrieval method as described above.

[0173] Embodiment 5

[0174] Based on the same inventive concept, Figure 5 , a first embodiment of the cross-modal privacy semantic retrieval system of the present invention is proposed, and the system may include:

[0175] The server 100 as described in the third embodiment;

[0176] The client 200 as described in the fourth embodiment;

[0177] The server 100 is connected to the client 200 for communication.

[0178] Combine the following Figure 5 The functional module schematic diagram shown in the figure provides a detailed description of the cross-modal privacy semantic retrieval system provided in this embodiment.

[0179] The server 100 may include:

[0180] The data receiving module 101 is used to receive a secret semantic representation vector sent by a client; wherein the secret semantic representation vector is obtained by extracting semantic features from multimodal data based on a multimodal joint representation model by the client, and encrypting the semantic representation vector;

[0181] The semantic association module 102 is used to determine the semantic search keywords associated with the dense semantic representation vector according to a preset search index table; wherein the preset search index table includes a mapping relationship between the semantic search keywords and the data addresses of the stored data;

[0182] The semantic retrieval module 103 is used to search the data address corresponding to the semantic retrieval keyword in the preset retrieval index table, obtain the encrypted semantic retrieval result, and send the encrypted semantic retrieval result to the client so that the client can decrypt and display the encrypted semantic retrieval result.

[0183] Furthermore, the semantic association module 102 may include:

[0184] A correlation calculation unit, used to determine the correlation value between the dense semantic representation vector and all search keywords in the preset search index table according to the preset search index table;

[0185] A semantic retrieval unit is used to determine semantic retrieval keywords associated with the dense semantic representation vector according to the association value; wherein the semantic retrieval keywords include retrieval keywords sorted according to the association value.

[0186] Furthermore, the first receiving module 101 is further used to receive the secret data to be stored and the corresponding secret keywords sent by the client; wherein the secret data to be stored is obtained by the client encrypting the data to be stored, and the secret keywords are obtained by the client extracting semantic features of the data to be stored based on the multimodal joint representation model and encrypting the keywords;

[0187] The server 100 may further include:

[0188] A storage module, used to store the secret data to be stored, and obtain the stored data and its data address;

[0189] The index table construction module is used to construct an index table according to the secret keywords and the data addresses of the stored data to obtain a preset retrieval index table.

[0190] Furthermore, the index table construction module may include:

[0191] An initial construction unit, used to construct an index table according to the secret key words and the data addresses of the stored data, and obtain a retrieval index table;

[0192] A similarity calculation unit, used to determine the similarity values ​​between the secret keywords according to the search index table;

[0193] A cluster processing unit, used to perform cluster processing according to the similarity value to obtain semantic search keywords; wherein the semantic search keywords include a cluster of dense keywords whose similarity value is less than a preset threshold;

[0194] A mapping unit is established to obtain a preset retrieval index table according to a mapping relationship between the semantic retrieval keywords and the corresponding data addresses of the stored data.

[0195] The client 200 may include:

[0196] A data acquisition module 201 is used to acquire multimodal data;

[0197] A feature extraction module 202 is used to extract semantic features from the multimodal data using a multimodal joint representation model to obtain a semantic representation vector;

[0198] An encryption module 203, used to encrypt the semantic representation vector to obtain a encrypted semantic representation vector;

[0199] The data sending module 204 is used to send the dense semantic representation vector to the server, so that the server determines the semantic search keyword associated with the dense semantic representation vector according to the preset search index table, searches the data address corresponding to the semantic search keyword in the preset search index table, obtains the dense semantic search result, and sends the dense semantic search result to the client; wherein the preset search index table includes the mapping relationship between the semantic search keyword and the data address of the stored data;

[0200] The result feedback module 205 is used to receive the encrypted semantic search result sent by the server, decrypt and display the encrypted semantic search result.

[0201] Furthermore, the feature extraction module 202 may include:

[0202] A preprocessing unit, configured to preprocess the multimodal data to obtain preprocessed multimodal data; wherein the preprocessing includes any one of format switching, text encoding, size scaling and noise elimination;

[0203] A semantic representation unit is used to input the preprocessed multimodal data into a multimodal joint representation model and output a semantic feature vector; wherein the multimodal joint representation model includes a representation learning model for performing modal conversion on the multimodal data.

[0204] It should be noted that the functions that can be implemented by each module in the cross-modal privacy semantic retrieval system provided in this embodiment and the corresponding technical effects achieved can refer to the description of the specific implementation methods in each embodiment of the cross-modal privacy semantic retrieval method of the present invention. For the sake of brevity of the specification, they will not be repeated here.

[0205] Embodiment 6

[0206] Based on the same inventive concept, this embodiment provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, etc., wherein a computer program is stored on the storage medium, and the computer program can be executed by one or more processors. When the computer program is executed by the processor, all or part of the steps of each embodiment of the cross-modal privacy semantic retrieval method of the present invention can be implemented.

[0207] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only optional embodiments of the present invention, and do not limit the patent scope of the present invention. All equivalent structures or equivalent process changes made by using the contents of the present specification and drawings under the inventive concept of the present invention, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A cross-modal privacy semantic retrieval method, characterized in that: Applied to a server, the method comprises: Receiving secret data to be stored and corresponding secret keywords sent by a client; wherein the secret data to be stored is obtained by encrypting the data to be stored by the client, and the secret keywords are obtained by encrypting the keywords after the client extracts semantic features of the data to be stored based on a multimodal joint representation model; Storing the secret data to be stored, and obtaining the data address of the stored data; Constructing an index table according to the secret keyword and the data address to obtain a preset search index table; Receiving a secret semantic representation vector sent by the client; wherein the secret semantic representation vector is obtained by encrypting the semantic representation vector after the client extracts semantic features from multimodal data based on a multimodal joint representation model to obtain the semantic representation vector; Determine the semantic search keywords associated with the dense semantic representation vector according to the preset search index table; wherein the preset search index table includes a mapping relationship between the semantic search keywords and the data addresses of the stored data; The data address corresponding to the semantic search keyword is searched in the preset search index table to obtain the encrypted semantic search result, and the encrypted semantic search result is sent to the client so that the client can decrypt and display the encrypted semantic search result.

2. The cross-modal privacy semantic retrieval method according to claim 1, characterized in that: The step of determining the semantic search keywords associated with the dense semantic representation vector according to the preset search index table includes: According to the preset search index table, determining the association value between the dense semantic representation vector and all search keywords in the preset search index table; According to the association value, semantic search keywords associated with the dense semantic representation vector are determined; wherein the semantic search keywords include search keywords sorted according to the association value.

3. The cross-modal privacy semantic retrieval method according to claim 1, characterized in that: The step of storing the secret data to be stored and obtaining the data address of the stored data comprises: The secret data to be stored is stored, and the stored data and its data address are obtained.

4. The cross-modal privacy semantic retrieval method according to claim 3, characterized in that: The step of constructing an index table according to the secret keyword and the data address to obtain a preset search index table comprises: According to the secret key words and the data addresses, an index table is constructed to obtain a retrieval index table; Determining the similarity values ​​between the secret keywords according to the search index table; According to the similarity value, clustering processing is performed to obtain semantic search keywords; wherein the semantic search keywords include a cluster of dense keywords whose similarity value is less than a preset threshold; A preset search index table is obtained according to the mapping relationship between the semantic search keywords and the corresponding data addresses of the stored data.

5. A cross-modal privacy semantic retrieval method, characterized in that: Applied to a client, the method comprises: Acquire multimodal data; Determining the multimodal data as data to be stored; Encrypting the data to be stored to obtain encrypted data to be stored; Extracting semantic features of the data to be stored based on a multimodal joint representation model to obtain keywords; Encrypting the keywords to obtain encrypted keywords; Sending the secret data to be stored and the secret keyword to a server, so that after receiving the secret data to be stored and the corresponding secret keyword, the server stores the secret data to be stored, obtains the data address of the stored data, and constructs an index table according to the secret keyword and the data address to obtain a preset retrieval index table; Extracting semantic features from the multimodal data using the multimodal joint representation model to obtain a semantic representation vector; Encrypting the semantic representation vector to obtain a encrypted semantic representation vector; The dense semantic representation vector is sent to the server, so that the server determines the semantic search keyword associated with the dense semantic representation vector according to the preset search index table, searches the data address corresponding to the semantic search keyword in the preset search index table, obtains the dense semantic search result, and sends the dense semantic search result to the client; wherein the preset search index table includes the mapping relationship between the semantic search keyword and the data address of the stored data; Receive the encrypted semantic retrieval result sent by the server, and decrypt and display the encrypted semantic retrieval result.

6. The cross-modal privacy semantic retrieval method according to claim 5, characterized in that: The step of extracting semantic features from the multimodal data using the multimodal joint representation model to obtain a semantic representation vector comprises: Preprocessing the multimodal data to obtain preprocessed multimodal data; wherein the preprocessing includes any one of format switching, text encoding, size scaling and noise elimination; The preprocessed multimodal data is input into the multimodal joint representation model to output a semantic feature vector; wherein the multimodal joint representation model includes a representation learning model for performing modality conversion on the multimodal data.

7. A server, characterized in that: The server includes a memory and a processor, and a cross-modal privacy semantic retrieval program is stored in the memory. When the cross-modal privacy semantic retrieval program is executed by the processor, the cross-modal privacy semantic retrieval method as described in any one of claims 1 to 4 is implemented.

8. A client, characterized in that: The client includes a memory and a processor, and a cross-modal privacy semantic retrieval program is stored in the memory. When the cross-modal privacy semantic retrieval program is executed by the processor, the cross-modal privacy semantic retrieval method as described in claim 5 or 6 is implemented.

9. A cross-modal privacy semantic retrieval system, characterized in that: The system comprises: The server according to claim 7; The client according to claim 8; The server is in communication connection with the client.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which can be executed by one or more processors to implement the cross-modal privacy semantic retrieval method as described in any one of claims 1 to 6.

Images