A method, apparatus, device and medium for processing unstructured data

Abstract

The application provides a processing method of unstructured data, comprising the following steps: performing word segmentation on unstructured data to obtain a word segmentation result, determining the weight of a sensitive word in the word segmentation result, and determining the weight of a non-sensitive word according to the similarity between the non-sensitive word and a privacy data attribute in the word segmentation result, and determining the privacy degree of the unstructured data through the weight of the sensitive word and the weight of the non-sensitive word. Since the non-sensitive word with a context relationship is considered, the method has high accuracy for privacy degree classification, and privacy protection processing based on the same has good privacy protection effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a non-structured data processing method and device, equipment and a computer readable storage medium. BACKGROUND

[0002] With the advent of the information age, data is growing explosively. Data can be divided into structured data and unstructured data. Structured data is data logically expressed and implemented in table structure, has a specific data format, and is usually stored and managed by a relational database. The privacy protection mechanism for structured data is quite mature, and for unstructured data, it is difficult to protect privacy due to the inability to represent it in a uniform structure.

[0003] The industry has proposed some privacy protection methods for unstructured data. For example, according to the proportion of the number of characters of private data or the number of private data in unstructured data, the privacy level of unstructured data is classified, and then a corresponding privacy protection mechanism is adopted based on the privacy level, such as desensitizing all private data in the text.

[0004] However, the above method of classifying privacy level based on the number of characters of private data or the number of private data has low accuracy, so that the privacy protection mechanism based on the privacy level is difficult to achieve good privacy protection effect. SUMMARY

[0005] The present application provides a non-structured data processing method, which regards the non-structured data as a whole, determines the privacy level of the non-structured data through the sensitive words and non-sensitive words in the non-structured data, has high accuracy, and can take corresponding privacy protection mechanism based on the privacy level to achieve good privacy protection effect. The present application also provides a device, equipment, computer readable storage medium and computer program product corresponding to the above method.

[0006] In a first aspect, the present application provides a method for processing unstructured data. The method can be implemented by a system for processing unstructured data. The system can be deployed in a cloud environment, an edge environment, or an end device (i.e., an end-side device). The cloud environment refers to a cluster of central computing devices owned by a cloud service provider and used to provide computing, storage, and communication resources. The edge environment refers to a cluster of edge computing devices that are geographically close to the end-side device and used to provide computing, storage, and communication resources. When the system is deployed in the cloud environment or the edge environment, the system can be provided to users in the form of a service. When the system is deployed in the end-side device, the system can be provided to users in the form of a client. In some implementations, the system for processing unstructured data includes multiple parts, which can be distributed in different environments.

[0007] Specifically, the system for processing unstructured data performs word segmentation on the unstructured data to obtain a segmentation result, determines a weight of a sensitive word in the segmentation result, determines a weight of a non-sensitive word in the segmentation result based on a similarity between the non-sensitive word and a privacy data attribute, and determines a privacy level of the unstructured data based on the weight of the sensitive word and the weight of the non-sensitive word.

[0008] The method considers not only the privacy data (i.e., the sensitive word) but also the non-sensitive word that has a contextual relationship with the sensitive word, and determines the privacy level of the unstructured data based on the sensitive word and the non-sensitive word, so that the method is more accurate and comprehensive in evaluating the privacy level. Further, the method can more accurately use a privacy protection mechanism corresponding to the level to perform privacy protection, and has a good privacy protection effect.

[0009] In some implementations, considering that the similarity between words can be measured by the distance between the words in a vector space, the system for processing unstructured data can further extract a word vector of the non-sensitive word and a word vector of the privacy data attribute, determine the similarity between the non-sensitive word and the privacy data attribute based on the distance between the word vector of the non-sensitive word and the word vector of the privacy data attribute, and determine the weight of the non-sensitive word based on the similarity between the non-sensitive word and the privacy data attribute.

[0010] The method introduces a method for calculating the similarity between words using word vectors in natural language processing, and uses the method to determine the similarity between the non-sensitive word and the privacy data attribute. Since the word vector preserves semantic features, the similarity determined based on the semantic features has high reliability.

[0011] In some implementations, the processing system of unstructured data can extract word vectors of the non-sensitive words and word vectors of the privacy data attributes by using a pre-trained word vector model. The word vectors are extracted by the word vector model with high efficiency and accuracy.

[0012] In some implementations, the definition of privacy data can be different in different application scenarios, and the language use and expression manner are quite different in different application scenarios, which makes the context of the same word quite different in the corpus of different application scenarios. If a general training corpus is used to train an initial word vector model, the accuracy of the trained word vector model can be not high. Based on this, the processing system of unstructured data can further obtain a training corpus matched with the application scenario of the unstructured data, and train an initial word vector model by using the training corpus to obtain a word vector model.

[0013] In some implementations, the same privacy data attribute corresponds to the words with similar context, but the privacy data words corresponding to the same privacy data attribute are always changing, for example, the privacy data words corresponding to the name can be "Zhang San", "Li Si", "Wang Wu", etc., and many privacy data words can appear very few times. The word vector model trained directly based on the training corpus is not accurate enough. In order to train a better word vector model and more accurately calculate the similarity to better assign the sensitive weight, the processing system of unstructured data can further preprocess the training corpus. Specifically, the sensitive words in the training corpus are identified, the sensitive words are replaced by the privacy data attributes of the sensitive words, and then the initial word vector model is trained by using the replaced training corpus to obtain a word vector model.

[0014] In some implementations, the processing system of unstructured data can further determine a privacy protection mechanism of the unstructured data according to the privacy degree of the unstructured data, and perform privacy protection on the unstructured data by using the privacy protection mechanism. This method can not only avoid the direct leakage of privacy information caused by privacy data, but also effectively prevent the indirect leakage of privacy information caused by semantic problems, thereby better protecting the privacy information.

[0015] In a second aspect, the present application provides a processing device of unstructured data. The device comprises:

[0016] A word segmentation module is configured to perform word segmentation on the unstructured data to obtain a word segmentation result.

[0017] A weight determination module is configured to determine the weight of the sensitive words in the word segmentation result, and determine the weight of the non-sensitive words according to the similarity between the non-sensitive words and the privacy data attributes in the word segmentation result.

[0018] A privacy degree determination module is configured to determine the privacy degree of the unstructured data according to the weight of the sensitive word and the weight of the non-sensitive word.

[0019] In some implementations, the weight determination module is specifically configured to:

[0020] extract a word vector of the non-sensitive word and a word vector of the privacy data attribute;

[0021] determine the similarity between the non-sensitive word and the privacy data attribute according to the distance between the word vector of the non-sensitive word and the word vector of the privacy data attribute;

[0022] determine the weight of the non-sensitive word according to the similarity between the non-sensitive word and the privacy data attribute.

[0023] In some implementations, the weight determination module is specifically configured to:

[0024] extract the word vector of the non-sensitive word and the word vector of the privacy data attribute by using a pre-trained word vector model.

[0025] In some implementations, the apparatus further comprises:

[0026] a communication module configured to obtain training corpus matching an application scenario of the unstructured data;

[0027] a training module configured to train an initial word vector model by using the training corpus to obtain a word vector model.

[0028] In some implementations, the apparatus further comprises:

[0029] a replacement module configured to identify a sensitive word in the training corpus and replace the sensitive word with a privacy data attribute;

[0030] The training module is specifically configured to:

[0031] train the initial word vector model by using the replaced training corpus to obtain the word vector model.

[0032] In some implementations, the apparatus further comprises:

[0033] a privacy protection processing module configured to determine a privacy protection mechanism of the unstructured data according to the privacy degree of the unstructured data and perform privacy protection on the unstructured data by using the privacy protection mechanism.

[0034] In a third aspect, the present application provides a device, the device comprising a processor and a memory. The processor and the memory are in communication with each other. The processor is configured to execute instructions stored in the memory, so that the device performs the method for processing unstructured data according to the first aspect or any implementation manner of the first aspect.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium storing instructions, the instructions instructing a device to perform the method for processing unstructured data according to the first aspect or any implementation manner of the first aspect.

[0036] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a device, cause the device to perform the method for processing unstructured data according to the first aspect or any implementation manner of the first aspect.

[0037] On the basis of the implementation manners of the aspects described above, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows.

[0039] Figure 1 An architecture diagram of a processing system of unstructured data provided by the embodiments of the present application;

[0040] Figure 2 An architecture diagram of a processing system of unstructured data provided by the embodiments of the present application;

[0041] Figure 3 A flowchart of a processing method of unstructured data provided by the embodiments of the present application;

[0042] Figure 4 A schematic diagram of determining the weight of a non-sensitive word provided by the embodiments of the present application;

[0043] Figure 5 A structural schematic diagram of a processing apparatus of unstructured data provided by the embodiments of the present application;

[0044] Figure 6 A structural schematic diagram of a device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0045] The terms "first", "second", etc. in the embodiments of the present application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features.

[0046] Firstly, some technical terms involved in the embodiments of the present application are introduced.

[0047] Unstructured data refers to data whose structure is irregular or incomplete, has no predefined data model, and is not convenient to use a two-dimensional logical table of a database to logically express and implement. The format of unstructured data is diverse. As an example, unstructured data can include various formats of documents or texts.

[0048] Word embedding is also called word embedding. Word embedding refers to a vector formed by mapping a word to a lower-dimensional continuous vector space. Word embedding can usually be represented by a sequence of real numbers. This representation of word embedding can be understood as a distributed representation based on a neural network, which retains the semantic features of the word.

[0049] For unstructured data such as personal resumes, medical records, and office documents, an industry has proposed a privacy protection method. Specifically, according to the definition of privacy data, privacy data existing in unstructured data is identified, the proportion of the number of privacy data characters is determined according to the ratio of the number of privacy data bits to the total number of unstructured data bits, or the proportion of the number of privacy data is determined according to the ratio of the number of privacy data to the total number of words in unstructured data, and the privacy level is classified by the proportion of the number of privacy data characters or the proportion of the number of privacy data. Then, based on the privacy level, a corresponding privacy protection mechanism is adopted, such as desensitizing all privacy data in unstructured data.

[0050] However, the above method of classifying privacy level based on the proportion of privacy data characters or the proportion of the number of privacy data ignores the relevance between contexts. In addition to privacy data, unstructured data such as personal resumes, medical records, and office documents can also include words that are highly similar to privacy data or have a strong direction to privacy data. Even if all privacy data in unstructured data is desensitized during privacy protection, some related information of privacy data can be inferred from these words that are highly similar to privacy data or have a strong direction to privacy data, resulting in incomplete desensitization of unstructured data and, to some extent, leakage of privacy information.

[0051] For example, in the privacy protection process of the sentence "My name is Wang Li, graduated from the University of Finance and Economics, and I don't want to work in X company.", if only the privacy data number ratio or the privacy data character number ratio is relied on to classify the privacy degree, and the privacy data is desensitized, the desensitized sentence becomes "My name is **, graduated from **, and I don't want to work in **."

[0052] The above privacy protection process only considers the privacy data, although the privacy data is masked to some extent, but the sentence desensitization is not complete. Specifically, since the semantic problem of the sentence is not considered, the semantic of the desensitized sentence is still complete, and the privacy degree of the sentence is not reduced to the minimum. Among them, "name" and privacy data name have a high degree of directionality; "graduated from" and privacy data school have a high degree of directionality; "stay" and "work" and privacy data work place have a high degree of directionality, and also express the will of the person; From these high directionality words, the related information of the desensitized privacy data or the will expressed by the person can be inferred.

[0053] Therefore, the accuracy of the above privacy degree classification method based on the privacy data number ratio or the privacy data character number ratio is not high, which leads to the privacy protection mechanism based on the privacy level being difficult to achieve good privacy protection effect.

[0054] Therefore, the accuracy of the above privacy degree classification method based on the privacy data number ratio or the privacy data character number ratio is not high, which leads to the privacy protection mechanism based on the privacy level being difficult to achieve good privacy protection effect.

[0055] The above-mentioned non-structured data processing method takes the non-structured data as a whole, not only considers the privacy data, i.e. sensitive words, but also considers the non-sensitive words having context relationship with the sensitive words, determines the privacy degree of the non-structured data based on the sensitive words and the non-sensitive words, so that the method is more accurate and more comprehensive in evaluating the privacy degree. Further, the method can more accurately adopt the corresponding level of privacy protection mechanism for privacy protection, and has good privacy protection effect.

[0056] As Figure 1As shown, the unstructured data processing system can be deployed in a cloud environment, specifically on one or more computing devices (e.g., a central server) within the cloud environment. The system can also be deployed in an edge environment, specifically on one or more computing devices (edge ​​computing devices), which can be servers. Furthermore, the system can be deployed on edge devices (i.e., end devices), including but not limited to desktops, laptops, smartphones, etc.

[0057] The cloud environment refers to a central computing device cluster owned by a cloud service provider, used to provide computing, storage, and communication resources; the edge environment refers to an edge computing device cluster that is geographically close to the end-side devices, used to provide computing, storage, and communication resources.

[0058] Edge devices can act as data providers, offering unstructured data so that unstructured data processing systems can process it to determine its privacy level and, based on that level, employ appropriate privacy protection mechanisms for privacy protection. Edge devices can provide unstructured data they generate or store for processing by the unstructured data processing system. In some implementations, edge devices can be network devices, such as terminal devices connected to a network; in this case, the edge device can obtain unstructured data from the network and provide it to the unstructured data processing system.

[0059] When an unstructured data processing system is deployed in a cloud or edge environment, it can be provided to users as a service. Specifically, users can access the cloud or edge environment through a browser, create an instance of the unstructured data processing system in the cloud or edge environment, and then interact with the instance of the unstructured data processing system through the browser to process the unstructured data.

[0060] Unstructured data processing systems can also be deployed on edge devices. Correspondingly, these systems can be provided to users as clients. Specifically, users run the client to process the unstructured data.

[0061] In some implementations, such as Figure 2 As shown, an unstructured data processing system comprises multiple parts (e.g., multiple subsystems, each containing multiple unit modules). Therefore, the various parts of the unstructured data processing system can be deployed in a distributed manner in different environments. For example, a portion of the unstructured data processing system can be deployed in three environments: a cloud environment, an edge environment, and an end device, or any two of these environments.

[0062] In order to make the technical solutions provided by the embodiments of the present application clearer and easier to understand, the processing method of unstructured data will be introduced from the perspective of the unstructured data processing system.

[0063] Referring to Figure 3 the flowchart of the processing method of unstructured data, the method comprises:

[0064] S302: The unstructured data processing system performs word segmentation on the unstructured data to obtain a word segmentation result.

[0065] In a specific implementation, the unstructured data processing system can use any one or more of a string matching-based word segmentation method, an understanding-based word segmentation method, and a statistical-based word segmentation method to perform word segmentation on the unstructured data to obtain a word segmentation result.

[0066] The string matching-based word segmentation method is to match a string to be analyzed with an entry in a machine dictionary according to a set strategy. If the string is found in the dictionary, the matching is successful, and a word is recognized. Then the matching operation is continuously performed, thereby achieving word segmentation on the unstructured data.

[0067] Further, the unstructured data processing system can also match in different directions when performing string matching, that is, the string matching-based word segmentation method can also be divided into a forward maximum matching method and a reverse maximum matching method. The unstructured data processing system can also perform limited matching according to different lengths when performing string matching, that is, the string matching-based word segmentation method can be divided into a longest matching method and a shortest matching method. In addition, the method can also be divided into a pure word segmentation method and an integrated method combining word segmentation and part-of-speech tagging according to whether it is combined with the part-of-speech tagging process.

[0068] The understanding-based word segmentation method is to simulate the understanding of a sentence to achieve the effect of recognizing words. Specifically, syntax analysis and semantic analysis are performed at the same time of word segmentation, and syntax information and semantic information are used to eliminate ambiguity, thereby achieving word segmentation on text and other unstructured data.

[0069] The statistical-based word segmentation method is to learn the rule of word segmentation by using a statistical machine learning model under the premise of a large amount of already segmented texts, so as to realize the segmentation of unknown texts. The statistical-based word segmentation method includes a maximum probability word segmentation method and a maximum entropy word segmentation method. The statistical model used in the above method includes one of an N-gram model, a Hidden Markov Model (HMM), a maximum entropy model (MEM), and a Conditional Random Fields (CRF).

[0070] Specifically, the unstructured data processing system can select a matched word segmentation method based on the language, scene, etc. of the unstructured data to perform word segmentation and obtain a word segmentation result.

[0071] In some implementations, in order to save storage space and improve the processing efficiency of unstructured data, the unstructured data processing system can also remove stop words after word segmentation to obtain a final word segmentation result.

[0072] S304: The unstructured data processing system determines the weight of the sensitive word in the word segmentation result, and determines the weight of the non-sensitive word according to the similarity between the non-sensitive word and the privacy data attribute in the word segmentation result.

[0073] Specifically, the unstructured data processing system can determine the sensitive word according to the word segmentation result, and the words other than the sensitive word in the word segmentation result are non-sensitive words. Then, the unstructured data processing system can determine the weight of the sensitive word, and determine the weight of the non-sensitive word according to the similarity between the non-sensitive word and the privacy data attribute. The weight is specifically used to measure the importance of the sensitive word or the non-sensitive word to the privacy degree of the entire unstructured data.

[0074] The privacy data attribute is used to describe the type of privacy data. For example, for the privacy data "Zhang San", the corresponding privacy data attribute is "name", and for the privacy data xx@yy.com, the corresponding privacy data attribute is "email address".

[0075] It is considered that the definition of privacy data may be different in different application scenarios. For example, for birthday or birthplace information, some application scenarios such as General Data Protection Regulation (GDPR) consider it as privacy, and some application scenarios such as medical scenarios do not consider it as privacy. As shown in the following table:

[0076] Table 1 Privacy data template in medical scenarios

[0077]

[0078]

[0079] Table 2 Privacy data template under GDPR scenario

[0080] I Name Is private XI Bank card number Is private II Email address Yes XII Nationality Yes III Mobile phone number Yes XIII Political party Yes IV Home phone number Yes XIV IP address Yes V Any address Yes XV GPS information Yes VI Identity card number Yes XVI DNA information Yes VII Passport number Yes XVII Fingerprint Yes VIII License plate number Yes XVIII Iris information Yes IX Date of birth Yes XIX Disease diagnosis Yes X Place of birth Yes

[0081] Based on this, when determining the sensitive word, the unstructured data processing system can match the attribute of each word in the word segmentation result with the privacy data attribute defined in the privacy data template under the current application scenario, so as to determine whether each word in the word segmentation result is a sensitive word or a non-sensitive word. The sensitive word and the non-sensitive word determined in this way have high accuracy.

[0082] Then, for the sensitive word, the unstructured data processing system can determine the weight of the sensitive word according to the set weight. For example, the weight of the sensitive word is set as a standard weight, such as 1, and the weight of the sensitive word can be obtained according to the set weight.

[0083] For the non-sensitive word, the weight of the non-sensitive word is determined according to the similarity between the non-sensitive word and the privacy data attribute. Specifically, the weight of the non-sensitive word is determined according to the correspondence between the similarity and the weight. The higher the similarity between the non-sensitive word and the privacy data attribute, the greater the weight of the non-sensitive word. The lower the similarity between the non-sensitive word and the privacy data attribute, the smaller the weight of the non-sensitive word.

[0084] For ease of understanding, the following will be described with a specific example. In this example, the unstructured data includes the sentence "My name is Zhang San", the unstructured data processing system determines "Zhang San" as a sensitive word and "name" as a non-sensitive word based on the privacy data attribute, and determines that the similarity between the non-sensitive word "name" and the privacy data attribute "name" is 0.9999 through calculation. According to the correspondence between the similarity and the weight ratio in advance, it can be determined that the weight ratio is 0.8, and thus it can be determined that the weight of "Zhang San" is 1 and the weight of "name" is 0.8.

[0085] S306: The unstructured data processing system determines the privacy degree of the unstructured data by the weight of the sensitive word and the weight of the non-sensitive word.

[0086] Specifically, the unstructured data processing system can obtain the privacy degree of the unstructured data by weighting and aggregating the weights of all sensitive words and all non-sensitive words.

[0087] In one example, the calculation formula of the privacy degree is as follows:

[0088]

[0089] wherein, the privacylevel represents the sensitive degree, also known as the sensitive level. n is the total number of sensitive words and non-sensitive words. i represents the sensitive value of the ith word in the unstructured data, and is specifically as follows:

[0090]

[0091] wherein, I i is the similarity between the ith word and the privacy data attribute when the ith word is a non-sensitive word. a i is the weight of the non-sensitive word, representing the influence degree of the non-sensitive word on the privacy degree of the unstructured data. wherein, a i is in the range of (0, 1), and is specifically determined according to the similarity between the non-sensitive word and the privacy data attribute.

[0092] In one example, the similarity between the non-sensitive word and the privacy data attribute and the weight of the non-sensitive word have the following corresponding relationship:

[0093]

[0094] The processing system of the unstructured data determines the weight of the non-sensitive word based on the above formula (3), and determines the privacy degree of the unstructured data based on the weight of the sensitive word and the weight of the non-sensitive word.

[0095] Based on the above description, the embodiment of the present application provides a processing method of unstructured data, which takes the unstructured data as a whole, considers the association relationship between contexts in the unstructured data, determines the weight of the sensitive word by using the similarity between the non-sensitive word and the privacy data attribute in the unstructured data, and determines the text privacy degree by the weight of the sensitive word and the weight of the non-sensitive word having the context relationship, which has higher accuracy.

[0096] Moreover, the method can more accurately determine the corresponding level of privacy protection mechanism. By using the privacy protection mechanism to protect the privacy of the unstructured data, it can not only avoid the direct leakage of privacy information caused by the privacy data, but also effectively prevent the indirect leakage of privacy information caused by semantic problems, so as to better protect the privacy information.

[0097] In order to verify that the privacy degree grading method proposed in the present application can better evaluate the privacy degree of unstructured data than the traditional method, an attack scene is designed for verification.

[0098] Specifically, in the attack scenario, the same obscuring and desensitizing processing is adopted for all the privacy data in the unstructured data of the text, specifically, all the privacy data is uniformly replaced with spaces, and then the privacy data is guessed by using the context vocabulary of the privacy data, the higher the probability of guessing the correct information, the more text privacy information the attacker can obtain, and the current privacy protection mechanism level is not enough, resulting in incomplete desensitization. Therefore, if the text privacy level classification is not accurate enough, it may lead to the use of a low-level privacy protection mechanism for desensitization of high-level text data, which makes the text data desensitization incomplete, and the desensitized text data may still leak privacy information.

[0099] The privacy level of the text can be verified by designing an attack scenario to predict privacy data. Accordingly, the privacy level of the text is calculated by using the privacy level classification method proposed in the embodiments of the present application and the traditional method respectively, and the ranking of the high and low levels of the text privacy level is performed. Which ranking is closer to the privacy level ranking obtained by using the attack scenario, indicates that the method can more accurately reflect the privacy level of the text.

[0100] Among them, the closeness of the ranking can be achieved by mean square error (MSE), and the calculation formula of MSE is as follows:

[0101]

[0102] Among them, n is the number of documents; x and y represent the lists of the two document privacy level rankings.

[0103] The embodiments of the present application provide the following experimental data:

[0104] Table 3: Privacy level ranking determined by different methods

[0105]

[0106] According to the ranking of Table 3, the MSE can be obtained:

[0107] MSE (verification ranking, ranking of the present application) = 6;

[0108] MSE (verification ranking, privacy data number ratio ranking) = 12;

[0109] MSE (verification ranking, privacy data character number ratio ranking) = 34.

[0110] Therefore, compared with the privacy degree grading method based on the proportion of privacy data or the proportion of privacy data character number, the ranking of the privacy degree grading method and verification method based on similarity proposed in the embodiment of the application is closer, and the method proposed in the embodiment of the application can more accurately grade the privacy degree of unstructured data.

[0111] Considering the semantic feature that there is relevance between context words, the embodiment of the application introduces a method of calculating word similarity by using word vectors in natural language processing (NLP), and uses the method to calculate the similarity between the non-sensitive word and the privacy data attribute.

[0112] Specifically, as shown in Figure 4 The unstructured data processing system can extract the word vector of the non-sensitive word and the word vector of the privacy data attribute, for example, input the non-sensitive word and the privacy data attribute into a pre-trained word vector model, so as to obtain the word vector of the non-sensitive word and the word vector of the privacy data attribute. Then, the similarity between the non-sensitive word and the privacy data attribute is determined according to the distance between the word vector of the non-sensitive word and the word vector of the privacy data attribute. Then, according to the similarity between the non-sensitive word and the privacy data attribute, the weight of the non-sensitive word is determined based on the corresponding relationship between the similarity and the weight (for example, the corresponding relationship shown in formula (3)).

[0113] The word vector model can be obtained by training through word2vec and the like. Specifically, the unstructured data processing system can construct an initial word vector model through word2vec, train the initial word vector model by using training corpus, and thus obtain the word vector model used for extracting the word vector.

[0114] Considering that the definition of privacy data can be different in different application scenarios, and the language use and expression manner of different application scenarios are quite different, which makes the context of the same word in the corpus of different application scenarios can be quite different, if the initial word vector model is trained by using the general training corpus, the accuracy of the trained word vector model can be not high. Based on this, the unstructured data processing system can obtain the training corpus matched with the application scenario of the unstructured data, and then train the initial word vector model by using the specific training corpus, and thus obtain the word vector model.

[0115] Further, even in the corpus of fixed application scenarios, the same privacy data attribute corresponds to the vocabulary with similar context, but the same privacy data attribute corresponds to the privacy data vocabulary that is always changing, for example, the privacy data vocabulary corresponding to the name can be "Zhang San", "Li Si", "Wang Wu", etc., and many privacy data vocabularies may appear very few times. The word vector model obtained by directly training based on the training corpus is not accurate enough. In order to train a better word vector model and more accurately calculate the similarity in order to better assign sensitive weights, the unstructured data processing system can also preprocess the training corpus. Specifically, sensitive words in the training corpus are identified, the privacy data attribute of the sensitive word is used to replace the sensitive word, and then the initial word vector model is trained using the replaced training corpus to obtain the word vector model.

[0116] The above is combined with Figures 1-4 The unstructured data processing method provided by the embodiments of the application is described in detail, and the device and equipment provided by the embodiments of the application will be introduced below with reference to the drawings.

[0117] Referring to Figure 5 The structure of the unstructured data processing device is shown in the structure diagram of the device 500, which includes:

[0118] The word segmentation module 502 is configured to perform word segmentation on the unstructured data to obtain a word segmentation result.

[0119] The weight determination module 504 is configured to determine the weight of the sensitive word in the word segmentation result, and determine the weight of the non-sensitive word according to the similarity between the non-sensitive word and the privacy data attribute in the word segmentation result.

[0120] The privacy degree determination module 506 is configured to determine the privacy degree of the unstructured data by the weight of the sensitive word and the weight of the non-sensitive word.

[0121] In some implementations, the weight determination module 504 is specifically configured to:

[0122] extract the word vector of the non-sensitive word and the word vector of the privacy data attribute;

[0123] determine the similarity between the non-sensitive word and the privacy data attribute according to the distance between the word vector of the non-sensitive word and the word vector of the privacy data attribute;

[0124] determine the weight of the non-sensitive word according to the similarity between the non-sensitive word and the privacy data attribute.

[0125] In some implementations, the weight determination module 504 is specifically configured to:

[0126] The word vectors of the non-sensitive words and the word vectors of the privacy data attributes are extracted using a pre-trained word vector model.

[0127] In some implementations, the device 500 further includes:

[0128] A communication module is used to acquire training corpus that matches the application scenario of the unstructured data;

[0129] The training module is used to train an initial word vector model using the training corpus to obtain the word vector model.

[0130] In some implementations, the apparatus further includes:

[0131] A replacement module is used to identify sensitive words in the training corpus and replace the sensitive words using privacy data attributes;

[0132] The training module is specifically used for:

[0133] The initial word vector model is trained using the replaced training corpus, resulting in the word vector model.

[0134] In some implementations, the apparatus further includes:

[0135] A privacy protection processing module is used to determine the privacy protection mechanism for the unstructured data based on the privacy level of the unstructured data, and to use the privacy protection mechanism to protect the privacy of the unstructured data.

[0136] The unstructured data processing apparatus 500 according to the embodiments of this application can correspond to executing the methods described in the embodiments of this application, and the above and other operations and / or functions of each module / unit of the unstructured data processing apparatus 500 are respectively for implementing Figure 3 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0137] This application embodiment also provides a device 600. The device 600 can be an edge device such as a laptop or desktop computer, or a computer cluster in a cloud or edge environment. The device 600 is specifically used to implement, for example... Figure 5 The function of the unstructured data processing device 500 in the illustrated embodiment.

[0138] Figure 6 A structural schematic diagram of a device 600 is provided, such as... Figure 6As shown, device 600 includes a bus 601, a processor 602, a communication interface 603, and a memory 604. The processor 602, memory 604, and communication interface 603 communicate via bus 601. Bus 601 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, Figure 6 The symbol is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. Communication interface 603 is used for external communication. For example, it can be used to acquire training corpora matching the application scenario of unstructured data, or to acquire unstructured data itself.

[0139] The processor 602 may be a central processing unit (CPU). The memory 604 may include volatile memory, such as random access memory (RAM). The memory 604 may also include non-volatile memory, such as read-only memory (ROM), flash memory, HDD, or SSD.

[0140] The memory 604 stores executable code, and the processor 602 executes the executable code to perform the aforementioned method for processing unstructured data.

[0141] Specifically, in achieving Figure 5 In the case of the illustrated embodiment, and Figure 5 When the modules of the unstructured data processing apparatus 500 described in the embodiment are implemented in software, the execution... Figure 5 The software or program code required for the functions of the word segmentation module 502, weight determination module 504, and privacy level determination module 506 are stored in the memory 604. The communication module function is implemented through the communication interface 603. The communication interface 603 receives unstructured data and transmits it to the processor 602 through the bus 601. The processor 602 executes the program code corresponding to each module stored in the memory 604, such as the program code corresponding to the word segmentation module 502, weight determination module 504, and privacy level determination module 506, to perform word segmentation of the unstructured data, then determine the weight of sensitive words, and determine the weight of non-sensitive words based on the similarity between non-sensitive words and privacy data attributes, and finally determine the privacy level of the unstructured data based on the weight of sensitive words and the weight of non-sensitive words.

[0142] Of course, the processor 602 can also execute the program code corresponding to the privacy protection processing module to perform the operation of determining the privacy protection mechanism of the unstructured data according to the privacy degree of the unstructured data, and performing privacy protection on the unstructured data by using the privacy protection mechanism.

[0143] The embodiment of the present application further provides a computer readable storage medium including instructions, which instruct a computer to execute the unstructured data processing method of the processing device 500 applied to unstructured data.

[0144] The embodiment of the present application further provides a computer readable storage medium including instructions, which instruct a computer to execute the unstructured data processing method of the processing device 500 applied to unstructured data.

[0145] The embodiment of the present application further provides a computer program product, which, when executed by a computer, causes the computer to execute any of the foregoing unstructured data processing methods. The computer program product can be a software installation package, and when any of the foregoing unstructured data processing methods needs to be used, the computer program product can be downloaded and executed on the computer.

Claims

1. A method for processing unstructured data, characterized in that, The method includes: The unstructured data is segmented to obtain segmentation results, which include sensitive words and non-sensitive words. Non-sensitive words are words in the segmentation results other than the sensitive words. The weights of sensitive words in the word segmentation results are determined, and the weights of non-sensitive words are determined based on the similarity between non-sensitive words and privacy data attributes in the word segmentation results. The weights of sensitive words are used to measure the importance of sensitive words to the privacy level of the unstructured data, and the weights of non-sensitive words are used to measure the importance of non-sensitive words to the privacy level of the unstructured data. The privacy data attributes are used to describe the type of privacy data. The privacy level of the unstructured data is determined by the weights of the sensitive words and the weights of the non-sensitive words. The step of determining the weight of the non-sensitive words based on the similarity between the non-sensitive words and privacy data attributes in the word segmentation results includes: Extract the word vectors of the non-sensitive words and the word vectors of the privacy data attributes; The similarity between the non-sensitive word and the privacy data attribute is determined based on the distance between the word vector of the non-sensitive word and the word vector of the privacy data attribute. The weight of the non-sensitive word is determined based on the similarity between the non-sensitive word and the privacy data attribute.

2. The method according to claim 1, characterized in that, The extraction of word vectors for the non-sensitive words and word vectors for the privacy data attributes includes: The word vectors of the non-sensitive words and the word vectors of the privacy data attributes are extracted using a pre-trained word vector model.

3. The method according to claim 2, characterized in that, The word vector model is trained in the following way: Obtain training corpus that matches the application scenarios of the unstructured data; The initial word vector model is trained using the training corpus to obtain the word vector model.

4. The method according to claim 3, characterized in that, The method further includes: Identify sensitive words in the training corpus and replace the sensitive words using privacy data attributes; The step of training an initial word vector model using the training corpus to obtain the word vector model includes: The initial word vector model is trained using the replaced training corpus, resulting in the word vector model.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The privacy protection mechanism for the unstructured data is determined based on the privacy level of the unstructured data; The privacy protection mechanism described above is used to protect the privacy of the unstructured data.

6. A device for processing unstructured data, characterized in that, The device includes: The word segmentation module is used to segment the unstructured data to obtain word segmentation results. The word segmentation results include sensitive words and non-sensitive words. The non-sensitive words are words in the word segmentation results other than the sensitive words. The weight determination module is used to determine the weight of sensitive words in the word segmentation results, and to determine the weight of non-sensitive words based on the similarity between non-sensitive words and privacy data attributes in the word segmentation results. The weight of the sensitive words is used to measure the importance of the sensitive words to the privacy level of the unstructured data, and the weight of the non-sensitive words is used to measure the importance of the non-sensitive words to the privacy level of the unstructured data. The privacy data attributes are used to describe the type of privacy data. A privacy level determination module is used to determine the privacy level of the unstructured data by the weights of the sensitive words and the weights of the non-sensitive words; Specifically, the weight determination module is used for: Extract the word vectors of the non-sensitive words and the word vectors of the privacy data attributes; The similarity between the non-sensitive word and the privacy data attribute is determined based on the distance between the word vector of the non-sensitive word and the word vector of the privacy data attribute. The weight of the non-sensitive word is determined based on the similarity between the non-sensitive word and the privacy data attribute.

7. The apparatus according to claim 6, characterized in that, The weight determination module is specifically used for: The word vectors of the non-sensitive words and the word vectors of the privacy data attributes are extracted using a pre-trained word vector model.

8. The apparatus according to claim 7, characterized in that, The device further includes: A communication module is used to acquire training corpus that matches the application scenario of the unstructured data; The training module is used to train an initial word vector model using the training corpus to obtain the word vector model.

9. The apparatus according to claim 8, characterized in that, The device further includes: A replacement module is used to identify sensitive words in the training corpus and replace the sensitive words using privacy data attributes; The training module is specifically used for: The initial word vector model is trained using the replaced training corpus, resulting in the word vector model.

10. The apparatus according to any one of claims 6 to 9, characterized in that, The device further includes: A privacy protection processing module is used to determine the privacy protection mechanism for the unstructured data based on the privacy level of the unstructured data, and to use the privacy protection mechanism to protect the privacy of the unstructured data.

11. A device, characterized in that, The device includes a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the device to perform the method as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, Includes instructions that instruct the device to perform the method as described in any one of claims 1 to 5.

Images