Cross-modal privacy semantic representation method, device, equipment and storage medium

Abstract

The present invention discloses a cross-modal privacy semantic representation method, device, equipment and storage medium, which relates to the field of data processing technology. The method includes: obtaining multimodal data; obtaining corresponding text data according to the multimodal data; extracting and encrypting keywords from the text data to obtain secret keywords; segmenting the preset knowledge graph according to the secret keywords to obtain secret subgraphs; embedding the secret subgraphs to obtain secret representation vectors corresponding to the secret keywords, so as to obtain the semantic representation results of the multimodal data. The present invention solves the problem of poor semantic correlation between secret keywords in the prior art, and achieves the effect of not only ensuring the semantic correlation between secret keywords, but also providing accurate semantic representation for subsequent privacy semantic retrieval.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cross-modal privacy semantic representation method, device, equipment and storage medium. Background Art

[0002] With the development of Internet technology and the popularization of cloud service technology, the contradiction between big data sharing and privacy protection has become increasingly fierce. Based on this, cross-modal data retrieval has become a rigid demand in the era of cloud services and big data, and the semantic representation of cross-modal data is a key component of the cross-modal data retrieval system.

[0003] Cross-modal semantic representation technology encodes data of different modalities through a model to obtain keywords, so that keywords of data of different modalities with the same semantics can have a high correlation and can be explicitly calculated. Cross-modal privacy semantic representation technology is a technology that adds privacy protection requirements on the basis of cross-modal semantic representation technology. This technology requires the retrieval system to be able to encode cross-modal data without uploading plaintext data to the cloud server, obtain confidential keywords, and then perform privacy semantic retrieval based on confidential keywords. However, the current cross-modal privacy semantic representation technology has the problem of poor semantic correlation between confidential keywords. Summary of the invention

[0004] The main purpose of the present invention is to provide a cross-modal privacy semantic representation method, device, equipment and storage medium, aiming to solve the technical problem of poor semantic correlation between confidential keywords in the prior art.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] In a first aspect, the present invention provides a cross-modal privacy semantic representation method, the method comprising:

[0007] Acquire multimodal data;

[0008] According to the multimodal data, obtaining corresponding text data;

[0009] Extracting and encrypting keywords from the text data to obtain encrypted keywords;

[0010] According to the dense keywords, the preset knowledge graph is segmented to obtain dense subgraphs;

[0011] The dense subgraph is embedded in a graph to obtain a dense representation vector corresponding to the dense keyword, so as to obtain a semantic representation result of the multimodal data.

[0012] Optionally, in the above cross-modal privacy semantic characterization method, the multimodal data includes data information of at least two different modalities;

[0013] The step of obtaining corresponding text data according to the multimodal data comprises:

[0014] When the multimodal data includes first modal data of a speech modality, converting the first modal data into first text data using speech recognition technology;

[0015] When the multimodal data includes second modality data of a video modality, converting the second modality data into second text data using a trained text generation model;

[0016] When the multimodal data includes third modal data in a text modality, the third modal data is directly determined as third text data.

[0017] Optionally, in the above cross-modal privacy semantic representation method, the step of extracting and encrypting keywords from the text data to obtain encrypted keywords includes:

[0018] Keyword extraction and encryption are performed on the first text data, the second text data and / or the third text data to obtain encrypted keywords.

[0019] Optionally, in the above cross-modal privacy semantic representation method, the step of extracting and encrypting keywords from the text data to obtain encrypted keywords includes:

[0020] Extract keywords from the text data using an unsupervised learning algorithm to obtain keywords;

[0021] The keywords are encrypted using a symmetric encryption algorithm to obtain encrypted keywords.

[0022] Optionally, in the above cross-modal privacy semantic representation method, the step of extracting keywords from the text data using an unsupervised learning algorithm to obtain keywords includes:

[0023] Performing word segmentation processing on the text data to obtain multiple words;

[0024] According to the plurality of words, a vocabulary network graph is drawn; wherein the network nodes of the vocabulary network graph correspond to the words, and the edge connecting two network nodes has an attribute value, and the attribute value is determined according to the co-occurrence relationship of the plurality of words;

[0025] According to the vocabulary network diagram, the multiple words are sorted and screened to obtain keywords representing the text data.

[0026] Optionally, in the above cross-modal privacy semantic representation method, before the step of segmenting the preset knowledge graph according to the secret keywords to obtain secret subgraphs, the method further includes:

[0027] Determine a basic knowledge graph through open source knowledge graph;

[0028] The basic knowledge graph is encrypted to obtain a preset knowledge graph; wherein the encryption algorithm used in the encryption process is consistent with the encryption algorithm used when encrypting the text data.

[0029] Optionally, in the above cross-modal privacy semantic representation method, the step of segmenting the preset knowledge graph according to the secret keywords to obtain a secret subgraph includes:

[0030] According to the secret keyword, matching entities corresponding to the secret keyword in the preset knowledge graph to obtain knowledge nodes;

[0031] In the preset knowledge graph, the knowledge node is taken as the center and divided according to the preset clipping distance to obtain a dense subgraph; wherein the length unit of the preset clipping distance is the edge between two entities, and the dense subgraph is a set of entities and edges within the preset clipping distance range centered on the knowledge node.

[0032] In a second aspect, the present invention provides a cross-modal privacy semantic representation device, the device comprising:

[0033] A data acquisition module, used for acquiring multimodal data;

[0034] A text description module, used to obtain corresponding text data according to the multimodal data;

[0035] A keyword extraction module, used to extract and encrypt keywords from the text data to obtain encrypted keywords;

[0036] A graph segmentation module, used to segment the preset knowledge graph according to the dense keywords to obtain dense subgraphs;

[0037] The graph embedding module is used to perform graph embedding on the dense subgraph to obtain a dense representation vector corresponding to the dense keyword to obtain a semantic representation result of the multimodal data.

[0038] In a third aspect, the present invention provides a cross-modal privacy semantic representation device, comprising a processor and a memory, wherein a cross-modal privacy semantic representation program is stored in the memory, and when the cross-modal privacy semantic representation program is executed by the processor, the cross-modal privacy semantic representation method as described above is implemented.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by one or more processors to implement the cross-modal privacy semantic representation method as described above.

[0040] The above one or more technical solutions provided by the present invention may have the following advantages or at least achieve the following technical effects:

[0041] The present invention proposes a cross-modal privacy semantic representation method, device, equipment and storage medium. After obtaining the corresponding text data according to the acquired multimodal data, the text data is subjected to keyword extraction and encryption to obtain secret keywords, which can protect data privacy and ensure data security; then the preset knowledge graph is segmented according to the secret keywords to obtain secret subgraphs, and the semantic information of the secret keywords is effectively expanded in the form of sub-knowledge graphs, which can more comprehensively express the semantic concepts of the secret keywords and maintain a strong correlation with the secret keywords; and the secret subgraph is embedded in the graph to obtain a secret representation vector corresponding to the secret keyword, thereby obtaining the semantic representation result of the multimodal data, encoding more abundant semantic information, and ensuring the semantic information relevance between the secret keywords. Under the premise of ensuring the privacy of user data, the present invention realizes the semantic representation of cross-modal data, which can not only ensure the semantic association between secret keywords, but also provide accurate semantic representation for subsequent privacy semantic retrieval. At the same time, the supported modal data can be dynamically increased or decreased according to business needs, which is more flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 It is a flowchart of the first embodiment of the cross-modal privacy semantic representation method of the present invention;

[0044] Figure 2 A schematic diagram of the hardware structure of a cross-modal privacy semantic representation device involved in the present invention;

[0045] Figure 3 It is a flowchart of the second embodiment of the cross-modal privacy semantic representation method of the present invention;

[0046] Figure 4 A schematic diagram of a basic knowledge graph in the second embodiment of the cross-modal privacy semantic representation method of the present invention;

[0047] Figure 5 A schematic diagram of a dense subgraph in the second embodiment of the cross-modal privacy semantic representation method of the present invention;

[0048] Figure 6 This is a schematic diagram of the functional modules of the first embodiment of the cross-modal privacy semantic representation device of the present invention.

[0049] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] It should be noted that in the present invention, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "includes..." does not exclude the existence of other identical elements in the process, method, article or system including the element. In addition, the meaning of "and / or" appearing in the full text includes three parallel schemes. Taking "A and / or B" as an example, it includes scheme A, or scheme B, or a scheme that satisfies both A and B.

[0052] In the present invention, if there is a description involving "first", "second", etc., the description of "first", "second", etc. is only for descriptive purposes and cannot be understood as indicating or suggesting its relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the present invention, the use of suffixes such as "module", "component" or "unit" used to represent elements is only to facilitate the description of the present invention and has no specific meaning in itself. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0053] For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention.

[0054] Analysis of existing technologies shows that with the development of Internet technology and the popularization of cloud service technology, the contradiction between big data sharing and privacy protection has intensified. Retrieval is a common function in big data scenarios, which is generally achieved through a set of retrieval systems. In the current context, it is very important for retrieval systems to ensure data privacy and security while improving data retrieval capabilities. At the same time, with the improvement of technology and the development of people's spiritual pursuits, people's concepts of privacy and data are constantly enriched. Privacy is no longer narrowly defined as traditional sensitive information such as identity information and financial information. Data such as life and travel are increasingly given the concept of privacy; data has also gradually evolved from traditional data concepts such as text and tables to the concept of coexistence of multiple modalities such as video, audio, and geography.

[0055] Based on this, the retrieval of cross-modal data has become a rigid demand in the era of cloud services and big data, and the semantic representation of cross-modal data is a key component of the cross-modal data retrieval system.

[0056] Cross-modal semantic representation technology is to jointly model data of different modalities. The resulting model can encode data of different modalities to obtain keywords, so that keywords of data of different modalities with the same semantics can have a high correlation and can be explicitly calculated. For example, the text modality "many people are meeting in the conference room" and the image modality of a conference room picture are respectively input into the model for encoding. The semantic correlation between the obtained text keywords and image keywords is higher than that of the keywords encoded with data of other semantic content.

[0057] Cross-modal privacy semantic representation technology is a technology that adds privacy protection requirements on the basis of cross-modal semantic representation technology. This technology requires the retrieval system to be able to encode cross-modal data without uploading plaintext data to the cloud server, obtain confidential keywords, and then perform privacy semantic retrieval based on the confidential keywords.

[0058] There are still some problems with the current cross-modal privacy semantic representation technology, such as:

[0059] 1. It is difficult to extract keywords from cross-modal data;

[0060] 2. It supports fewer modes and is difficult to expand;

[0061] 3. The semantic correlation between dense keywords is poor.

[0062] In view of the technical problem that the semantic correlation between secret keywords is poor in the prior art, the present invention provides a cross-modal privacy semantic representation method, and the overall idea is as follows:

[0063] Acquire multimodal data; obtain corresponding text data based on the multimodal data; perform keyword extraction and encryption on the text data to obtain dense keywords; segment the preset knowledge graph based on the dense keywords to obtain dense subgraphs; perform graph embedding on the dense subgraphs to obtain dense representation vectors corresponding to the dense keywords, so as to obtain semantic representation results of the multimodal data.

[0064] Through the above technical scheme, after obtaining the corresponding text data according to the acquired multimodal data, the text data is subjected to keyword extraction and encryption to obtain secret keywords, which can protect data privacy and ensure data security; then the preset knowledge graph is segmented according to the secret keywords to obtain secret subgraphs, and the semantic information of secret keywords is effectively expanded in the form of sub-knowledge graphs, which can more comprehensively express the semantic concepts of secret keywords and maintain a strong correlation with secret keywords; and the secret subgraph is embedded in the graph to obtain a secret representation vector corresponding to the secret keyword, thereby obtaining the semantic representation result of multimodal data, encoding richer semantic information, and ensuring the semantic information relevance between secret keywords. Under the premise of ensuring the privacy of user data, the present invention realizes the semantic representation of cross-modal data, which can not only ensure the semantic association between secret keywords, but also provide accurate semantic representation for subsequent retrieval of privacy semantics. At the same time, the supported modal data can be dynamically increased or decreased according to business needs, which is more flexible.

[0065] The cross-modal privacy semantic representation method, apparatus, device and storage medium provided by the present invention are described in detail below with reference to the accompanying drawings through specific embodiments and implementation methods.

[0066] Embodiment 1

[0067] Reference Figure 1The flowchart of the present invention proposes the first embodiment of the cross-modal privacy semantic representation method, which is applied to a cross-modal privacy semantic representation device. The cross-modal privacy semantic representation device refers to a terminal device or a network device that can achieve network connection, which can be a terminal device such as a mobile phone, a computer, a tablet computer, or a network device such as a server or a cloud platform. The method can also be applied to a semantic retrieval system including a terminal device and a network device. When the method is applied to a semantic retrieval system, part of the steps of the method can be performed on the terminal device. After the obtained results are sent to the network device, the remaining steps continue to be performed on the network device.

[0068] like Figure 2 , which is a schematic diagram of the hardware structure of a cross-modal privacy semantic representation device. The device may include: a processor 1001, such as a CPU (Central Processing Unit), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Those skilled in the art can understand that Figure 2 The hardware structure shown in does not constitute a limitation on the cross-modal privacy semantic characterization device of the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0069] Specifically, the communication bus 1002 is used to realize the connection and communication between these components;

[0070] The user interface 1003 is used to connect to the client and perform data communication with the client. The user interface 1003 may include an output unit, such as a display screen, an input unit, such as a keyboard, and optionally, the user interface 1003 may also include other input / output interfaces, such as a standard wired interface and a wireless interface;

[0071] The network interface 1004 is used to connect to the backend server and perform data communication with the backend server. The network interface 1004 may include an input / output interface, such as a standard wired interface, a wireless interface, such as a Wi-Fi interface;

[0072] The memory 1005 is used to store various types of data, which may include, for example, instructions of any application or method in the cross-modal privacy semantic representation device, and data related to the application. The memory 1005 may be a high-speed RAM memory or a stable memory, such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the processor 1001.

[0073] For details, please refer to Figure 2, the memory 1005 may include an operating system, a network communication module, a user interface module, and a cross-modal privacy semantic representation program, wherein the network communication module is mainly used to connect to a server and perform data communication with the server;

[0074] The processor 1001 is used to call the cross-modal privacy semantic characterization program stored in the memory 1005, and perform the following operations:

[0075] Acquire multimodal data;

[0076] According to the multimodal data, obtaining corresponding text data;

[0077] Extracting and encrypting keywords from the text data to obtain encrypted keywords;

[0078] According to the dense keywords, the preset knowledge graph is segmented to obtain dense subgraphs;

[0079] The dense subgraph is embedded in a graph to obtain a dense representation vector corresponding to the dense keyword, so as to obtain a semantic representation result of the multimodal data.

[0080] Based on the above cross-modal privacy semantic representation device, the following is combined with Figure 1 The flowchart shown in the figure describes in detail the cross-modal privacy semantic representation method of this embodiment. The method may include the following steps:

[0081] Step S110: Acquire multimodal data.

[0082] Multimodal data refers to data information with multiple different modes, such as video data, audio data, image data, text data, and other modal data. Users can manually input through the user interface of the device, or receive data transmitted by other devices through the network interface of the device. The data obtained here can include data of multiple modes, or only one mode. Here, only data including multiple modes is used as an example for explanation.

[0083] Step S120: Obtain corresponding text data according to the multimodal data.

[0084] For the multimodal data obtained in step S110, when it contains data in non-text modalities such as images, audio or video, it is necessary to perform a modal conversion to convert the data from non-text modality to text modality. When it contains data in text modality, it is not necessary to perform a modal conversion, and the data in the text modality is directly used as the text data finally obtained in this step. When the non-text modal data is modally converted, the model trained by combining the characteristics of the non-text modal data such as pictures, audio or video can be used to perform text recognition, and the text describing the non-text modal data can be obtained, which can improve the accuracy of the text description of the non-text modal data.

[0085] In the specific implementation process, when the multimodal data includes image data, image recognition technology can be used to obtain text data corresponding to the image data, and a pre-trained deep learning model can be used for text description; when the multimodal data includes audio data, speech recognition technology can be used to obtain text data corresponding to the audio data, and a deep learning model such as Deep Speech V2 can be used for text description; when the multimodal data includes video data, text description generation technology can be used to obtain text data corresponding to the video data, and a pre-trained text generation model such as VideoBERT can be used for text description. It should be noted that no matter how many modal data the multimodal data includes, after converting the non-text modal data into text modal data, all text modal data, that is, data that is originally in text modality and converted text modal data, are summarized to obtain text data corresponding to the acquired multimodal data.

[0086] Step S130: extracting and encrypting keywords from the text data to obtain encrypted keywords.

[0087] Since the text data may contain many words or unnecessary words that may easily affect the accuracy of semantic representation, keywords are extracted from the text data; and since the privacy of the data must be guaranteed, the extracted keywords are encrypted to obtain the secret keywords of the multimodal data. Among them, the keyword extraction can be performed by the Text Rank algorithm, and the extracted keywords can also be encrypted by a symmetric encryption algorithm, such as the DES (Data Encryption Standard) algorithm, the RC (Rivest Cipher, stream encryption) algorithm, the BlowFish algorithm (block cipher algorithm), etc. It should be noted that when the method of this embodiment is performed on a terminal device or a network device, the secret keywords can be processed subsequently to obtain the semantic representation results of the cross-modal data. When the method of this embodiment is performed on a semantic retrieval system including a terminal device and a network device, after the terminal device executes steps S110 to S130, the obtained secret keywords can be sent to the network device. After receiving the secret keywords, the network device can process the secret keywords to obtain the semantic representation results of the cross-modal data, wherein the sending and receiving processes are encrypted, so the privacy of the data can still be guaranteed.

[0088] Step S140: Segment the preset knowledge graph according to the dense keywords to obtain dense subgraphs.

[0089] The preset knowledge graph is a graph structure that visually describes the vocabulary entities in the form of nodes and the relationships between entities in the form of edges, which can explicitly describe the association between words. The preset knowledge graph is an encrypted open source knowledge graph such as Wikidata, wherein the encryption method and key are consistent with the encryption method and key for encrypting the keywords in step S30, which can ensure that after the key is matched, the secret keywords are processed and the privacy of the data is guaranteed. According to the secret keywords, the preset knowledge graph is segmented by matching the secret keywords with the nodes in the preset knowledge graph, and according to the set clipping distance, a sub-knowledge graph centered on the secret keywords is obtained, which includes the entity corresponding to the secret keyword, other entities within the set clipping distance around the entity, and a set of edges representing the association relationship between these entities. Since the preset knowledge graph is in an encrypted state, the sub-knowledge graph obtained by clipping is also in an encrypted state, that is, a secret subgraph is obtained. Among them, the number of secret subgraphs is consistent with the number of secret keywords, that is, the secret subgraph is one-to-one corresponding to the secret keywords.

[0090] Here, a sub-knowledge graph centered on secret keywords is used as the expression of secret keywords, and entities and relationships that are highly correlated with secret keywords are used to jointly represent secret keywords. This can not only express the semantic concepts of secret keywords more comprehensively, but also accurately represent the semantic concepts of secret keywords through other related entities and corresponding relationships, making easily confused secret keywords more distinguishable.

[0091] Step S150: embedding the dense subgraph to obtain a dense representation vector corresponding to the dense keyword to obtain a semantic representation result of the multimodal data.

[0092] Each dense keyword can correspond to a dense subgraph. After the dense subgraph is obtained in step S140, the dense subgraph is embedded based on the random walk algorithm to obtain its dense representation vector, that is, the dense representation vector corresponding to the dense keyword. The vector is the semantic representation of the corresponding dense keyword. After obtaining the corresponding dense subgraphs for all dense keywords, and performing graph embedding operations respectively, and obtaining the dense representation vectors, a vector set can be obtained after summarizing, that is, the semantic representation result of the multimodal data of this embodiment. The semantic representation result can be directly used for the input of semantic retrieval, so as to obtain the retrieval result, thereby realizing the retrieval of cross-modal data. For example, the user can directly input multimodal data on the device. After obtaining the semantic representation result through the method of this embodiment, the semantic representation result is input into the retrieval system, and the final cross-modal retrieval result can be obtained after retrieval.

[0093] The cross-modal privacy semantic representation method provided in this embodiment obtains the corresponding text data according to the acquired multimodal data, extracts and encrypts the text data to obtain confidential keywords, which can protect data privacy and ensure data security; then, the preset knowledge graph is segmented according to the confidential keywords to obtain confidential subgraphs, and the semantic information of the confidential keywords is effectively expanded in the form of sub-knowledge graphs, which can more comprehensively express the semantic concepts of the confidential keywords and maintain a strong correlation with the confidential keywords; and the confidential subgraph is embedded in the graph to obtain the confidential representation vector corresponding to the confidential keyword, thereby obtaining the semantic representation result of the multimodal data, encoding more abundant semantic information, and ensuring the semantic information relevance between the confidential keywords. Under the premise of ensuring the privacy of user data, the present invention realizes the semantic representation of cross-modal data, which can not only ensure the semantic association between confidential keywords, but also provide accurate semantic representation for subsequent privacy semantic retrieval. At the same time, the supported modal data can be dynamically increased or decreased according to business needs, which is more flexible.

[0094] Embodiment 2

[0095] Based on the same inventive concept, Figures 3 to 5 , a second embodiment of the cross-modal privacy semantic representation method of the present invention is proposed. The method is applied to a cross-modal privacy semantic representation device. The method can be further applied to a cross-modal privacy semantic retrieval system running on the device. After the system uses the method to perform privacy semantic representation on multimodal data, it performs privacy semantic retrieval based on the semantic representation results to obtain semantic retrieval results. Semantic retrieval is a process of automatically querying and extracting relevant information from an information source at a semantic level according to requirements by correctly analyzing the grammatical format and understanding the relationship between semantics and vocabulary. Privacy semantic retrieval can be based on privacy data, and requires the security of privacy data.

[0096] Combine the following Figure 3 The flowchart shown in the figure describes in detail the cross-modal privacy semantic representation method of this embodiment. The method may include the following steps:

[0097] Step S210: Acquire multimodal data.

[0098] Specifically, the multimodal data includes data information of at least two different modes. Here, the type of modal data in the multimodal data is not limited, and the amount of modal data is not limited. The type of modal data and the amount of modal data can be increased or decreased according to business needs. This embodiment is illustrated by multimodal data including three modal data of voice, video, and text, and English is used as an example language.

[0099] Step S220: Obtain corresponding text data according to the multimodal data.

[0100] In the specific implementation process, the text generation model can be used to obtain corresponding text data according to the multimodal data. Specifically, the multimodal data is used as the input of the text generation model, and then each modal data is input into the corresponding text generation model to output the text modal data respectively, thereby obtaining text data describing the multimodal data.

[0101] Specifically, step S220 may include:

[0102] Step S221: when the multimodal data includes first modal data of a speech modality, the first modal data is converted into first text data using speech recognition technology.

[0103] Among them, speech recognition technology is a technology that converts data containing human language sounds into text and constructs a mapping relationship between text and speech. Modal conversion of non-text modal data is the key to semantic representation and retrieval of cross-modal data. In this embodiment, the Deep Speech V2 model can be used as a speech recognition model. The model can use speech data of a set language for model training. Here, English speech data is used for model training to obtain a speech recognition model. During the implementation process, the first modal data is used as the input of the model, and the corresponding text description is directly output.

[0104] Step S222: When the multimodal data includes second modality data of a video modality, the second modality data is converted into second text data using a trained text generation model.

[0105] The trained text generation model is trained based on a data set that has both text descriptions and video content as training data. During implementation, the trained text generation model is used, the second modality data is used as input of the model, and the corresponding text description is directly output.

[0106] In this embodiment, the text generation model adopts the Video BERT model, and the Video BERT model is trained using the BERT model in the language model and the self-supervised learning training method. When training the text generation model, a data set including text description data and video content data is used as training data. The training method is as follows: first, a feature vector is extracted from the video content data, and then the feature vector is discretized by a clustering method to construct a visual vocabulary, and then combined with the text description data to form a cross-modal vocabulary; then visual tags (tokens) and linguistic tags are derived from the cross-modal vocabulary and input into the text generation model to be trained. The model learns the bidirectional joint distribution on these tag sequences, thereby constructing a mapping relationship between visual tags and linguistic tags, and obtaining a trained text generation model.

[0107] Step S223: When the multimodal data includes third modal data in a textual mode, directly determining the third modal data as third textual data.

[0108] The text data itself is text modal data and does not need to be processed. The next step can be performed directly. The next step may include aggregating the converted text data and text modal data to obtain the text data, and then extracting and encrypting keywords from the text data. It may also be directly extracting and encrypting keywords from the converted text data or text modal data. The specific operation depends on the type and amount of multimodal data and is set according to actual conditions.

[0109] In this embodiment, the multimodal data of the three modal data of voice, video, and text are processed according to the above steps, and the three text data are summarized to obtain text data. Each modal data is unified into a text modality, without a limit on the number of modalities or the type of modalities. When new modal data is added, only the corresponding text description generation model needs to be added. This method is extensible.

[0110] Step S230: extracting and encrypting keywords from the text data to obtain encrypted keywords.

[0111] In one embodiment, step 230 may include:

[0112] Step 231: extract and encrypt keywords from the first text data, the second text data and / or the third text data to obtain encrypted keywords.

[0113] Since the modal data types of multimodal data can be increased or decreased and can be multiple, there will be multiple corresponding text data. When there are multiple text data, such as the three text data in this embodiment, some or all of them can be summarized and directly subjected to keyword extraction and encryption processing to obtain the encrypted keywords corresponding to the multimodal data.

[0114] In another embodiment, step 230 may include:

[0115] Step 232: extract keywords from the text data using an unsupervised learning algorithm to obtain keywords.

[0116] Keywords are generally single words or phrases composed of multiple words, which refer to general words or phrases that can reflect the theme or meaning of the text. In this embodiment, the Text Rank algorithm is used to extract text keywords. The algorithm is an unsupervised algorithm that can take a single document or text data as input. The algorithm principle is to split the text into words as network nodes to form a vocabulary network graph, and regard the correlation between words as a recommendation or voting relationship, so that the importance of each word can be calculated, and then the top N words are screened as keywords representing the entire document or the entire text data.

[0117] Specifically, step 232 may include:

[0118] Step 232.1: Perform word segmentation processing on the text data to obtain multiple words.

[0119] During the specific implementation process, the text data can be segmented, POS tagged, and stop words can be removed. During word segmentation, the stuttering word segmentation is adopted to retain words of seven parts of speech, including common nouns, proper nouns, common verbs, adverbs, noun verbs, adjectives, and adverbs, and finally multiple words can be obtained.

[0120] In this embodiment, the text data obtained in step S220 is segmented to obtain multiple words.

[0121] Step 232.2: Draw a vocabulary network diagram based on the multiple words; wherein the network nodes of the vocabulary network diagram correspond to the words, and the edges connecting two network nodes have attribute values, and the attribute values ​​are determined according to the co-occurrence relationship of the multiple words.

[0122] In the specific implementation process, a word is taken as a network node, and a word network diagram of multiple words is drawn. In the diagram, the edges between network nodes, that is, between words, have attribute values, and the attribute values ​​are determined according to the co-occurrence relationship of the words represented by the two network nodes.

[0123] In this embodiment, a vocabulary network diagram is drawn based on the multiple words obtained in step S232.1, wherein the network node set of the vocabulary network diagram is composed of the multiple words, and the edge between the two network nodes is determined by analyzing the co-occurrence relationship between the words represented by any two network nodes in the network node set, that is, the edge between the two network nodes is drawn only when the words corresponding to the two network nodes co-occur in a window of length K, wherein K represents the window size, i.e., a maximum of K words co-occur.

[0124] Step 232.3: Sort and filter the multiple words according to the vocabulary network diagram to obtain keywords representing the text data.

[0125] In the vocabulary network diagram, the weight of each network node, i.e., each vocabulary, is calculated according to an iterative algorithm until convergence; then the weight of each vocabulary is sorted, and a preset number of vocabulary is screened, and this preset number of vocabulary is the keyword representing the text data. In actual implementation, the sorting method and the preset number can be set according to actual conditions.

[0126] Step 233: Encrypt the keyword using a symmetric encryption algorithm to obtain a secret keyword.

[0127] The keywords obtained in step S232.3 are encrypted by using a symmetric encryption algorithm. This algorithm has a small amount of calculation, a fast encryption speed, and a high encryption efficiency. It can achieve high-speed encryption and decryption processing, and can use a long key, which is difficult to crack. It can ensure the privacy and security of multimodal data and can also improve the processing speed of the method.

[0128] Step S240: Obtain a preset knowledge graph.

[0129] Specifically, step S240 may include:

[0130] Step S241: determining a basic knowledge graph through an open source knowledge graph;

[0131] A knowledge graph is a knowledge base in which knowledge is integrated through a graph-structured data model or topology. The graph structure is used to visualize knowledge entities in the form of nodes, and the relationships between knowledge entities in the form of edges, thereby explicitly describing the associations between knowledge.

[0132] like Figure 4 The figure shows a schematic diagram of the preset knowledge graph of this embodiment. In the figure, there are multiple entities, and one entity represents a node. In this embodiment, based on the aforementioned example language set as English, the open source knowledge graph Wikidata is used as the basic knowledge base. The open source knowledge graph is a large database that stores a large amount of information in Wikipedia and Freebase, and has the ability to describe common things, which can meet the needs of this embodiment.

[0133] Step S242: encrypting the basic knowledge graph to obtain a preset knowledge graph; wherein the encryption algorithm used in the encryption process is consistent with the encryption algorithm used when encrypting the text data.

[0134] The basic knowledge graph is encrypted to ensure the privacy of the data. It should be noted that the encryption method used here is consistent with the encryption method used to encrypt the keywords in step S233, and the keys used are also consistent, so that subsequent secret keywords can be successfully matched with the preset knowledge graph.

[0135] Using symmetric encryption methods to encrypt text keywords and knowledge graphs can ensure data privacy and security while maintaining the matching ability of confidential keywords.

[0136] Step S250: Segment the preset knowledge graph according to the dense keywords to obtain dense subgraphs.

[0137] Specifically, step S250 may include:

[0138] Step S251: According to the secret keyword, matching the entity corresponding to the secret keyword in the preset knowledge graph to obtain a knowledge node;

[0139] According to the secret keyword, entity matching is performed in the preset knowledge graph, that is, the entity corresponding to the secret keyword is searched in the preset knowledge graph, and the entity is determined as a knowledge node. When there are multiple secret keywords, multiple knowledge nodes can be obtained, and these knowledge nodes may have no association or association in the preset knowledge graph.

[0140] In this embodiment, taking a secret keyword as an example, Figure 4 In the knowledge graph shown, entity 1 is used to represent the knowledge node corresponding to the secret keyword.

[0141] Step S252: In the preset knowledge graph, with the knowledge node as the center, the graph is divided according to the preset clipping distance to obtain a dense subgraph; wherein the length unit of the preset clipping distance is the edge between two entities, and the dense subgraph is a set of entities and edges within the preset clipping distance range with the knowledge node as the center.

[0142] Specifically, with the knowledge node as the center, the sub-knowledge graph is segmented according to the preset clipping distance to obtain the dense subgraph corresponding to the dense keyword, that is, the set of entities and edges within the preset clipping distance range centered on the knowledge node. Each dense keyword can obtain a corresponding dense subgraph. Figure 4 As shown, the length unit of the preset clipping distance is the edge R between two entities, that is, the association relationship between the two entities, and each edge is represented by a different sequence as R1, R2, ..., Rm. Figure 4 In the knowledge graph, entity 1 is taken as the center and segmented according to the preset clipping distance to obtain the following Figure 5 The dense state subgraph diagram is shown in Figure 1. Figure 5 (a) represents the dense subgraph obtained when the clipping distance is 1 unit. During clipping, entity 1 is taken as the center, and the entities and edges that can be reached by only one edge with entity 1 are split. The set of entities and edges obtained is the dense subgraph of the dense keyword represented by entity 1; Figure 5 (b) represents the dense subgraph obtained when the clipping distance is 2 units. When clipping, entity 1 is taken as the center, and the entities and edges that can be reached with two or fewer edges are split. The set of entities and edges obtained is the dense subgraph of the dense keyword represented by entity 1. Similarly, Figure 5 (c) shows the dense subgraph obtained when the cropping distance is 3 units.

[0143] Using a sub-knowledge graph centered on secret keywords as the expression of secret keywords, and using entities and relationships that are highly correlated with secret keywords to jointly represent secret keywords, can not only express the semantic concepts of secret keywords more comprehensively, but also accurately represent the semantic concepts of secret keywords through other related entities and relationships, making easily confused secret keywords more distinguishable, and better retaining the semantic relevance of multimodal data encoding under the premise of ensuring user data privacy.

[0144] Step S260: embedding the dense subgraph to obtain a dense representation vector corresponding to the dense keyword to obtain a semantic representation result of the multimodal data.

[0145] Each secret keyword can be represented as a sub-knowledge graph. Based on this sub-knowledge graph, a graph embedding operation is performed on it to obtain its secret representation vector, which is the semantic representation of the secret keyword. Among them, the random walk algorithm (Deep Walk algorithm) can be used for graph embedding operation.

[0146] In this embodiment, the Deep Walk algorithm is used for graph embedding operation. The algorithm includes two parts: generation and update. The random walk generator is used to generate a random path similar to a sentence; the random walk update procedure is used to input the random path into the Skip-Gram model to obtain the hidden representation of the node in the dense subgraph. The specific implementation process is that the random walk generator samples a point on the dense subgraph using uniform distribution as the starting point of the random path, and then arbitrarily determines a current point. The next point is obtained by sampling from all neighbors of the current point using uniform distribution, and then the point is determined as the current point. Repeat until the maximum length is set and stop. It can also stop when the current point has no neighbors. After stopping, all points are connected, which is the random path obtained by the random walk generator. The length of each random path may be different, but the longest will not exceed the set maximum length. According to this method, a random path is generated for each node in the dense subgraph. After the random path is generated, the random walk updater treats it as a sentence, inputs it into the Skip-Gram model, calculates the objective function, and updates the hidden representation of the nodes in the dense subgraph. The objective function is:

[0147] j(Φ)=-logP r (u k |v j ; Φ),

[0148] Among them, v j represents the jth point on the random path, u k Indicates that the dense subgraph does not include vj The node P r (u k |v j Φ) means that at a given v j Under the condition of k The conditional probability of occurrence, Φ represents a trainable parameter.

[0149] The method of this embodiment achieves privacy semantic representation of cross-modal data under the premise of ensuring user data privacy and modal data relevance.

[0150] For more implementation details of the implementation of steps S210 to S260, reference may be made to the description of the implementation based on steps S110 to S150 in the first embodiment, which will not be repeated here for the sake of brevity.

[0151] The cross-modal privacy semantic representation method provided in this embodiment can not only ensure the semantic association between confidential keywords, but also the supported modalities can be dynamically increased or decreased according to business needs, thereby ensuring the flexibility and semantic accuracy of subsequent privacy semantic retrieval, thereby ensuring the accuracy of the retrieval, which is of great significance to meeting user privacy security and cross-modal data retrieval needs.

[0152] Embodiment 3

[0153] Based on the same inventive concept, Figure 6 , a first embodiment of a cross-modal privacy semantic representation device of the present invention is proposed. The cross-modal privacy semantic representation device can be a virtual device applied to a cross-modal privacy semantic representation device.

[0154] Combine the following Figure 6 The functional module diagram shown in FIG. 1 is a schematic diagram of a cross-modal privacy semantic representation device provided by this embodiment, and the device may include:

[0155] A data acquisition module, used for acquiring multimodal data;

[0156] A text description module, used to obtain corresponding text data according to the multimodal data;

[0157] A keyword extraction module, used to extract and encrypt keywords from the text data to obtain encrypted keywords;

[0158] A graph segmentation module, used to segment the preset knowledge graph according to the dense keywords to obtain dense subgraphs;

[0159] The graph embedding module is used to perform graph embedding on the dense subgraph to obtain a dense representation vector corresponding to the dense keyword to obtain a semantic representation result of the multimodal data.

[0160] Further, the multimodal data includes data information of at least two different modalities; the text description module may include:

[0161] a first data processing unit, configured to convert first modal data in a speech modality into first text data by using a speech recognition technology when the multimodal data includes first modal data in a speech modality;

[0162] a second data processing unit, configured to, when the multimodal data includes second modal data of a video modality, convert the second modal data into second text data by using a trained text generation model;

[0163] The third data processing unit is used to directly determine the third modal data as third text data when the multimodal data includes third modal data in text mode.

[0164] Furthermore, the keyword extraction module may include:

[0165] The first keyword extraction unit is connected to the first data processing unit, the second data processing unit and / or the third data processing unit, and is used to extract and encrypt the first text data, the second text data and / or the third text data to obtain encrypted keywords.

[0166] Furthermore, the keyword extraction module may include:

[0167] A keyword extraction submodule, used to extract keywords from the text data through an unsupervised learning algorithm to obtain keywords;

[0168] The encryption submodule is used to encrypt the keywords through a symmetric encryption algorithm to obtain encrypted keywords.

[0169] Furthermore, the keyword extraction submodule may include:

[0170] A splitting unit, used for performing word segmentation processing on the text data to obtain a plurality of words;

[0171] A drawing unit, used for drawing a vocabulary network diagram according to the plurality of words; wherein the network nodes of the vocabulary network diagram correspond to the words, and the edges connecting two network nodes have attribute values, and the attribute values ​​are determined according to the co-occurrence relationship of the plurality of words;

[0172] The screening unit is used to sort and screen the multiple words according to the vocabulary network diagram to obtain keywords representing the text data.

[0173] Furthermore, the device may also include:

[0174] A preset knowledge graph acquisition module is used to determine a basic knowledge graph through an open source knowledge graph; encrypt the basic knowledge graph to obtain a preset knowledge graph; wherein the encryption algorithm used in the encryption process is consistent with the encryption algorithm used when encrypting the text data.

[0175] Furthermore, the atlas segmentation module may include:

[0176] A matching unit, configured to match entities corresponding to the secret keywords in the preset knowledge graph according to the secret keywords to obtain knowledge nodes;

[0177] A segmentation unit is used to segment the preset knowledge graph according to a preset clipping distance with the knowledge node as the center to obtain a dense subgraph; wherein the length unit of the preset clipping distance is the edge between two entities, and the dense subgraph is a collection of entities and edges within the preset clipping distance range with the knowledge node as the center.

[0178] It should be noted that the functions that can be realized by each module in the cross-modal privacy semantic representation device provided in this embodiment and the corresponding technical effects achieved can refer to the description of the specific implementation methods in each embodiment of the cross-modal privacy semantic representation method of the present invention. For the sake of brevity of the specification, they will not be repeated here.

[0179] Embodiment 4

[0180] Based on the same inventive concept, Figure 2 , which is a schematic diagram of the hardware structure of the cross-modal privacy semantic representation device involved in each embodiment of the present invention. This embodiment provides a cross-modal privacy semantic representation device, which may include a processor and a memory, wherein a cross-modal privacy semantic representation program is stored in the memory, and when the cross-modal privacy semantic representation program is executed by the processor, all or part of the steps of each embodiment of the cross-modal privacy semantic representation method of the present invention are implemented.

[0181] Specifically, the cross-modal privacy semantic representation device refers to a terminal device or network device that can achieve network connection, which can be a terminal device such as a mobile phone, computer, tablet computer, portable computer, or a network device such as a server or cloud platform.

[0182] It can be understood that the device may also include a communication bus, a user interface and a network interface.

[0183] The communication bus is used to realize the connection and communication between these components.

[0184] The user interface is used to connect to the client and communicate data with the client. The user interface may include an output unit, such as a display screen, and an input unit, such as a keyboard. Optionally, the user interface may also include other input / output interfaces, such as a standard wired interface and a wireless interface.

[0185] The network interface is used to connect to the backend server and perform data communication with the backend server. The network interface may include an input / output interface, such as a standard wired interface, or a wireless interface, such as a Wi-Fi interface.

[0186] The memory is used to store various types of data, which may include, for example, instructions of any application or method in the cross-modal privacy semantic representation device, and data related to the application. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. Optionally, the memory can also be a storage device independent of the processor.

[0187] The processor is used to call the cross-modal privacy semantic representation program stored in the memory and execute the cross-modal privacy semantic representation method as described above. The processor can be an application specific integrated circuit (Application Specific Integrated Circuit, referred to as ASIC), a digital signal processor (Digital Signal Processor, referred to as DSP), a digital signal processing device (Digital Signal Processing Device, referred to as DSPD), a programmable logic device (Programmable Logic Device, referred to as PLD), a field programmable gate array (Field Programmable Gate Array, referred to as FPGA), a controller, a microcontroller, a microprocessor or other electronic components, used to execute all or part of the steps of each embodiment of the cross-modal privacy semantic representation method as described above.

[0188] Embodiment 5

[0189] Based on the same inventive concept, this embodiment provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., wherein a computer program is stored on the storage medium, and the computer program can be executed by one or more processors. When the computer program is executed by the processor, all or part of the steps of each embodiment of the cross-modal privacy semantic representation method of the present invention can be implemented.

[0190] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0191] The above descriptions are only optional embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent structures or equivalent process changes made using the contents of the present invention's specification and drawings, or directly or indirectly applied in other related technical fields, are included in the patent protection scope of the present invention.

Claims

1. A cross-modal privacy semantic representation method, characterized in that: The method comprises: Acquire multimodal data; According to the multimodal data, obtaining corresponding text data; Extracting and encrypting keywords from the text data to obtain encrypted keywords; According to the dense keywords, the preset knowledge graph is segmented to obtain dense subgraphs; Performing graph embedding on the dense subgraph to obtain a dense representation vector corresponding to the dense keyword to obtain a semantic representation result of the multimodal data; The step of segmenting the preset knowledge graph according to the dense keywords to obtain dense subgraphs includes: According to the secret keyword, matching the entity corresponding to the secret keyword in the preset knowledge graph to obtain a knowledge node; In the preset knowledge graph, the knowledge node is taken as the center and divided according to the preset clipping distance to obtain a dense subgraph; wherein the length unit of the preset clipping distance is the edge between two entities, and the dense subgraph is a set of entities and edges within the preset clipping distance range centered on the knowledge node.

2. The cross-modal privacy semantic representation method according to claim 1, characterized in that: The multimodal data includes data information of at least two different modalities; The step of obtaining corresponding text data according to the multimodal data comprises: When the multimodal data includes first modal data of a speech modality, converting the first modal data into first text data using speech recognition technology; When the multimodal data includes second modality data of a video modality, converting the second modality data into second text data using a trained text generation model; When the multimodal data includes third modal data in a text modality, the third modal data is directly determined as third text data.

3. The cross-modal privacy semantic representation method according to claim 2, characterized in that: The step of extracting and encrypting keywords from the text data to obtain encrypted keywords includes: Keyword extraction and encryption are performed on the first text data, the second text data and / or the third text data to obtain encrypted keywords.

4. The cross-modal privacy semantic representation method according to claim 1, characterized in that: The step of extracting and encrypting keywords from the text data to obtain encrypted keywords includes: Extract keywords from the text data using an unsupervised learning algorithm to obtain keywords; The keywords are encrypted using a symmetric encryption algorithm to obtain encrypted keywords.

5. The cross-modal privacy semantic representation method according to claim 4, characterized in that: The step of extracting keywords from the text data by an unsupervised learning algorithm to obtain keywords comprises: Performing word segmentation processing on the text data to obtain multiple words; According to the plurality of words, a vocabulary network graph is drawn; wherein the network nodes of the vocabulary network graph correspond to the words, and the edge connecting two network nodes has an attribute value, and the attribute value is determined according to the co-occurrence relationship of the plurality of words; According to the vocabulary network diagram, the multiple words are sorted and screened to obtain keywords representing the text data.

6. The cross-modal privacy semantic representation method according to claim 1, characterized in that: Before the step of segmenting the preset knowledge graph according to the dense keywords to obtain dense subgraphs, the method further includes: Determine a basic knowledge graph through open source knowledge graph; The basic knowledge graph is encrypted to obtain a preset knowledge graph; wherein the encryption algorithm used in the encryption process is consistent with the encryption algorithm used when encrypting the text data.

7. A cross-modal privacy semantic representation device, characterized in that: The device comprises: A data acquisition module, used for acquiring multimodal data; A text description module, used to obtain corresponding text data according to the multimodal data; A keyword extraction module, used to extract and encrypt keywords from the text data to obtain encrypted keywords; A graph segmentation module, used to segment the preset knowledge graph according to the dense keywords to obtain dense subgraphs; A graph embedding module, used for performing graph embedding on the dense subgraph to obtain a dense representation vector corresponding to the dense keyword, so as to obtain a semantic representation result of the multimodal data; The atlas segmentation module comprises: A matching unit, configured to match entities corresponding to the secret keywords in a preset knowledge graph according to the secret keywords to obtain knowledge nodes; A segmentation unit is used to segment the preset knowledge graph according to a preset clipping distance with the knowledge node as the center to obtain a dense subgraph; wherein the length unit of the preset clipping distance is the edge between two entities, and the dense subgraph is a collection of entities and edges within the preset clipping distance range with the knowledge node as the center.

8. A cross-modal privacy semantic representation device, characterized in that: The cross-modal privacy semantic representation device includes a memory and a processor, and a cross-modal privacy semantic representation program is stored in the memory. When the cross-modal privacy semantic representation program is executed by the processor, the cross-modal privacy semantic representation method as described in any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program can be executed by one or more processors to implement the cross-modal privacy semantic representation method as described in any one of claims 1 to 6.

Images