Generalized processing and protection method for text data in economic big data

Abstract

The present application relates to the field of data generalization technology, and specifically to a generalization processing protection method for text data in economic big data, the method comprising: obtaining text data; training an isolation forest using a text attribute vector, determining the intra-class deviation value of a feature vector based on the intra-class difference between feature vectors; determining the content complexity of a cluster based on the degree of distribution dispersion of the intra-class deviation value, and determining the content convertibility between clusters in combination with the degree of similarity of feature vectors within the cluster; determining the voting weight of an isolation tree based on the content convertibility, and obtaining an anomaly score of a text attribute vector; and screening text data that needs generalization processing based on the anomaly score to obtain a generalization processing result. The present application optimizes the isolation forest algorithm, improves the effect of generalization processing, and reduces the risk of text data in economic big data being attacked by analyzing the differences in text data content under the same content and the characteristics of interchangeable combinations appearing in economic big data from different data sources.

Description

Technical Field

[0001] The present application relates to the technical field of data generalization, and in particular to a generalized processing and protection method for text data in economic big data. Background Art

[0002] In the era of digital economy, big data analysis can help analyze market demand, optimize resource allocation, supply chain management, precision marketing and other fields, and promote the effective use of economic big data. However, attention should be paid to privacy and data security protection during the analysis process.

[0003] Economic big data is a type of multi-data source, containing massive amounts of data, including digital data, text data and other types of data. Text data usually comes from news websites, annual reports, monthly reports, research reports, etc. It is one of the important data resources in economic big data. The sensitivity and importance of text data in economic big data are self-evident. Once leaked or tampered with, it may cause serious security problems in the production of related companies. Therefore, in order to avoid the security of data being threatened by attackers, it is usually necessary to protect the text data and enhance the security of text data in economic big data.

[0004] Data generalization is a process of blurring and distorting specific information, which helps reduce the risk of important information in text data being leaked. However, generalization will reduce the availability of text data. Therefore, the text data for data generalization often needs to be pre-set. Since economic big data changes over time, the content of text data will be constantly updated. When the pre-set text data is inappropriate, there may be a problem of reduced security of the text data. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a generalized processing and protection method for text data in economic big data to solve the existing problems.

[0006] The generalized processing protection method for text data in economic big data of this application adopts the following technical solution:

[0007] An embodiment of the present application provides a generalized processing protection method for text data in economic big data, the method comprising the following steps:

[0008] Acquire text data of various types of content in different fields from multiple data sources;

[0009] According to the text data in each category of content in each field, a text attribute vector of each text data is obtained through the PV-DM model, an isolation forest model is trained through the text attribute vector of the text data, and an intra-class deviation value of each feature vector in a cluster is determined based on the intra-class difference between the feature vectors obtained by dimensionality reduction of the text attribute vectors;

[0010] Determine the content complexity of each cluster based on the distribution discreteness of the intra-class deviation value of the feature vector in each cluster; determine the content convertibility between two clusters based on the content complexity combined with the similarity of the feature vectors in the two clusters;

[0011] Determine the voting weight of the isolated tree based on the content convertibility between the clusters where the feature vectors in the tree nodes are located, and obtain the abnormality score of each text attribute vector based on the voting weight;

[0012] Based on the abnormal scores of the text attribute vectors corresponding to all text data in each field and each type of content, the text data that needs to be generalized is screened, and the generalization processing results are obtained using the generalization model.

[0013] Preferably, the isolation forest model comprises:

[0014] The text data of each type of content obtained in each field is taken as input, and the PV-DM model is used to obtain the text attribute vector of each text data of each type of content obtained in each field;

[0015] The length of each text attribute vector is used as the dimension of each sample in the isolation forest algorithm, and the maximum depth of the isolation tree is determined by cross-validation.

[0016] The text attribute vectors of all text data of each category of content obtained in each field are used as training samples to train the isolation forest model.

[0017] Preferably, the intra-class deviation value of each feature vector in the cluster is determined based on the intra-class difference between the feature vectors obtained by dimensionality reduction of the text attribute vector, including:

[0018] Calculate the cosine similarity between any two text attribute vectors in each tree node on each isolated tree, input all the obtained cosine similarities into the Otsu threshold algorithm, use the Otsu threshold algorithm to obtain the segmentation threshold of the similarity, and record the combination of two text attribute vectors with cosine similarity greater than or equal to the segmentation threshold as a true similar combination;

[0019] The PCA principal component analysis method is used to reduce each text attribute vector to a feature vector of a preset dimension, and the clustering algorithm is used to divide the feature vectors obtained by reducing the dimension of all text attribute vectors in each tree node into multiple clusters;

[0020] Each eigenvector in each cluster is taken as a calculation object, and the absolute value of the difference in cosine similarity between each calculation object, each eigenvector and the same eigenvector is calculated respectively. The mean of the accumulation results of the absolute value of the difference on all eigenvectors in each cluster is taken as the intra-class deviation value of the calculation object.

[0021] Preferably, determining the content complexity of each cluster based on the distribution discreteness of the intra-class deviation value of the feature vector in each cluster includes:

[0022] The distribution variance and range of the intra-class deviation value of the feature vector in each cluster are calculated respectively, and the product of the distribution variance and the range is used as the content complexity of each cluster.

[0023] Preferably, the determining the content convertibility between two clusters based on the content complexity combined with the similarity of feature vectors in the two clusters includes:

[0024] The mean of the cosine similarity between two feature vectors in two clusters is taken as the similarity feature value between the two clusters.

[0025] The sum of the content complexity of the two clusters is used as the numerator;

[0026] The ratio of the molecule to the similarity feature value is taken as the content convertibility between the two clusters.

[0027] Preferably, the step of determining the voting weight of the isolated tree based on the content convertibility between the clusters where the feature vectors in the tree nodes are located, and obtaining the abnormality score of each text attribute vector based on the voting weight includes:

[0028] The minimum spanning tree corresponding to each tree node is determined by using the content convertibility between clusters obtained by clustering the feature vectors corresponding to all text attribute vectors in each tree node, and the ratio of the sum of the connection weights between the nodes in the minimum spanning tree to the number of nodes is used as the sample segmentation accuracy of the tree node;

[0029] The voting weight of each isolated tree is determined based on the sample segmentation accuracy of all tree nodes in each isolated tree;

[0030] The text attribute vector of the text data of each category of content obtained in each field is taken as input, and the abnormality score of each text attribute vector is obtained using the trained isolation forest model based on the voting weights of all isolated trees.

[0031] Preferably, the method of determining the minimum spanning tree corresponding to each tree node by using the content convertibility between clusters obtained by clustering the feature vectors corresponding to all text attribute vectors in each tree node comprises:

[0032] Each cluster obtained by clustering all feature vectors obtained by dimensionality reduction of text data in each tree node is regarded as a node, and the content convertibility between two clusters is used as the connection weight between the corresponding two nodes. Based on all clusters obtained by clustering, the feature graph of each tree node is obtained;

[0033] The feature map of each tree node is taken as input, and the Prim algorithm is used to generate the minimum spanning tree of the feature map of each tree node.

[0034] Preferably, the step of determining the voting weight of an isolated tree based on the sample segmentation accuracy of all tree nodes in each isolated tree comprises:

[0035] The tree nodes in the same layer of the isolated tree are recorded as the tree nodes of the same layer, and the maximum sample segmentation accuracy among the tree nodes of the same layer is taken as the segmentation representative value of this layer;

[0036] Sort the segmentation representative values ​​of each layer of the isolation tree from small to large according to the depth of the corresponding layer to obtain a segmentation representative sequence;

[0037] For each adjacent element in the segmentation representative sequence, calculate the difference between the next element and the previous element in the adjacent elements, and determine the calculation result of the logarithmic function with the sum of the difference and a preset constant as a true number and a natural constant as a base;

[0038] The average of the accumulated results of the calculation results on the segmentation representative sequence is used as the voting weight of each isolated tree.

[0039] Preferably, the method of screening text data that needs generalization processing based on the abnormal scores of the text attribute vectors corresponding to all text data in each field and each type of content includes:

[0040] The anomaly scores of the text attribute vectors corresponding to all text data of each category of content in each field are taken as input, and the data clustering algorithm is used. The number of clusters is set to 2, and the output is 2 clusters;

[0041] The mean of all anomaly scores in each cluster is calculated, and the text data corresponding to the anomaly scores in the cluster with the largest mean is recorded as a high-difference sample of each category of content in each field, and the high-difference sample is taken as an object that needs to be generalized.

[0042] Preferably, the obtaining of a generalized processing result by using a generalized model includes:

[0043] For each high-difference sample of each category of content in each field, respectively calculate the edit distance between each high-difference sample and high-difference samples of all categories of content in the remaining fields, and take the set of high-difference samples of all categories of content in the remaining fields whose edit distance is less than the distance threshold as the anonymous candidate set of each high-difference sample;

[0044] Each high-difference sample and the anonymous candidate set of each high-difference sample are used as inputs of the k-anonymity algorithm, and the k-anonymity algorithm is used to obtain the generalization processing result of each high-difference sample.

[0045] This application has at least the following beneficial effects:

[0046] This application aims to achieve generalized processing of text data with different contents in economic big data from different fields, so as to achieve the protection of text data. First, text data of multiple categories of content in different fields are obtained from multiple data sources, which is convenient for subsequent targeted training of multiple isolation forest models and screening of text data that need generalization processing; secondly, the PCA algorithm is used to reduce the dimension of the text attribute vector of each text data. Data dimension reduction can ensure the accuracy of the isolation forest after the high-precision isolation forest is obtained by training with high-latitude text attribute data; data dimension reduction on this basis can reduce the amount of calculation when processing large amounts of text data in economic big data; then, the intra-class deviation value of each feature vector in the clustering cluster is determined based on the intra-class difference between the feature vectors, so as to achieve the difference evaluation of the text data content under each category of content; then, the content complexity of each clustering cluster is determined and combined with the similarity of the feature vectors in the two clustering clusters. The degree of content convertibility between two clusters is determined, and the possibility of combinable conversion between text data contents of different data sources under the same content is evaluated through the content convertibility, so as to facilitate the subsequent analysis of the segmentation accuracy of each tree node; then, the voting weight of the isolated tree is determined based on the content convertibility between the clusters where the feature vectors in the tree nodes are located, and the calculation method of the voting weight is optimized through the differences of the samples in the tree nodes; finally, based on the abnormal scores of the text attribute vectors corresponding to all text data under each type of content in each field, the text data that needs to be generalized is screened to obtain the generalized processing results, avoiding the problem of low text data availability and security caused by inappropriate pre-set text data for data generalization, improving the effect of generalization processing of text data, and ensuring the security of text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A flowchart of the generalized processing and protection method for text data in economic big data provided in this application. DETAILED DESCRIPTION

[0049] In order to further explain the technical means and effects adopted by this application to achieve the predetermined invention purpose, the following is a detailed description of the generalized processing protection method for text data in economic big data proposed in this application, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0050] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0051] The following is a detailed description of the specific scheme of the generalized processing and protection method for text data in economic big data provided by the present application in conjunction with the accompanying drawings.

[0052] An embodiment of the present application provides a generalized processing and protection method for text data in economic big data.

[0053] Specifically, the following generalized processing and protection methods for text data in economic big data are provided, please refer to Figure 1 , the method comprises the following steps:

[0054] Step S001, obtaining text data of multiple types of content in different fields from multiple data sources.

[0055] In this application, text data in economic big data is obtained from data sources in multiple different fields, including but not limited to news websites, corporate official websites, statistical data, and research reports. Among them, the collected text data content is M category, including but not limited to corporate annual reports, quarterly reports, market analysis reports, and user reviews.

[0056] It should be noted that the data source of the text data collected in the economic big data and the content of the text data can be set by the implementer, and this application does not impose any special restrictions on this.

[0057] In this application, text data from N fields of economic big data are obtained, including but not limited to commodity export, scientific research, education, and industrial production. At the same time, in order to avoid errors caused by transmission and other conditions during data acquisition, Apache Spark technology is used to clean the acquired data. Apache Spark technology is a well-known technology, and the specific process will not be repeated.

[0058] Step S002, training an isolation forest based on the text attribute vectors of the text data in each category of content in each field, and determining the intra-class deviation value of each feature vector in the cluster based on the intra-class differences between the feature vectors obtained by dimensionality reduction of the text attribute vectors.

[0059] The data content of economic big data may be different in different fields, so there may be large differences in the data content between the text data collected in various fields. Attackers may crack the text data based on this difference.

[0060] For example, in the field of commodity exports, the volume of imported and exported goods, the increase or decrease of tariffs, and the trade volume are the contents that appear frequently in text data; while in other fields, such as scientific research and education, the focus is on technological development and educational technology, and the relevant contents of the above-mentioned words appear relatively less frequently in the text data of scientific research and teaching. Therefore, once an attacker obtains a piece of text data in which the volume of imported and exported goods, the increase or decrease of tariffs, and the trade volume appear frequently in economic big data, the attacker can easily infer that this text data belongs to the text data in the economic big data of the commodity export field, which poses a serious threat to the information security of the production and export of related enterprises or manufacturers.

[0061] Generalization processing helps reduce the risk of sensitive information being leaked, but since there is a lot of text data in each field and they have certain correlations, it is impossible to accurately determine the text data that needs to be generalized.

[0062] Therefore, in this application, firstly, based on the degree of difference distinction of text data in various fields, generalization processing is performed on text data with high difference distinction, so as to achieve security protection of text data in economic big data in each field and reduce the risk of attack on the text data in various fields.

[0063] Based on the above analysis, the differences between text data in each field and each type of content can be analyzed using isolation forests. The specific process is as follows:

[0064] First, in order to facilitate the difference measurement of text data, the text data is vectorized. For the text data of each type of content obtained in each field, as an embodiment of the present application, the PV-DM (Distributed Memory Model of Paragraph Vectors) model is used to convert the text data of each type of content obtained in each field into data that can be processed by the isolation forest. Specifically, the text data of each type of content obtained in each field is used as input, the text vector length in the PV-DM model is set to 300, the window size is set to 5, and the output is the text attribute vector of each text data of each type of content obtained in each field. Among them, the PV-DM model is a well-known technology, and the specific process will not be repeated.

[0065] Furthermore, the text attribute vectors of each text data of each type of content obtained in each field are obtained respectively. Secondly, the length of each text attribute vector is used as the dimension of each sample in the isolation forest algorithm, and the maximum depth of the isolated tree is determined by cross-validation. The number of isolated trees in the isolation forest is set to 100, and the number of samples extracted each time is 128. The number of isolated trees and the number of samples extracted can be set by the implementer according to the actual situation. Among them, the isolation forest algorithm and the cross-validation method are both well-known technologies, and the specific process will not be repeated.

[0066] For the text data of the a-th category content, if the text data of the a-th category content obtained in a certain field has a high degree of difference, then when the text attribute vector of the text data of the a-th category content obtained in the field is used as a sample to train the isolation forest, for each tree node on each isolated tree, the deeper the depth of the tree node in the isolated tree, the higher the similarity between the isolated tree nodes at the same depth, the higher the accuracy of segmentation, and the better the effect of the isolation forest algorithm.

[0067] Take the jth tree node in the i-th isolated tree in the isolation forest As an example, the cosine similarity between any two text attribute vectors in the tree node is calculated, and all the obtained cosine similarities are input into the Otsu threshold algorithm. The Otsu threshold algorithm is used to obtain the segmentation threshold of the similarity, and the combination of two text attribute vectors whose cosine similarity is greater than or equal to the segmentation threshold is recorded as a true similar combination.

[0068] Secondly, the PCA principal component analysis algorithm is used to reduce the dimensionality of each text attribute vector. The input of the PCA algorithm is each text attribute vector, and the output is a feature vector reduced to v dimensions, where v takes an empirical value of 5. The PCA algorithm is a well-known technology and the specific process will not be repeated here.

[0069] It should be noted that the purpose of vector dimensionality reduction is to ensure the accuracy of the isolation forest after training with high-precision isolation forest using high-dimensional text attribute data; data dimensionality reduction on this basis can reduce the amount of computation when processing large amounts of text data in economic big data.

[0070] Furthermore, the v-dimensional feature vectors obtained by reducing the dimension of each text attribute vector are used for the tree nodes All feature vectors obtained by dimensionality reduction of the text data are divided into multiple clusters using a clustering algorithm, and it is considered that the feature vectors in each cluster are obtained by text attribute vectors with high similarity. Among them, vector clustering is a commonly used technology in the field of data processing, including but not limited to k-means clustering algorithm, mean shift clustering algorithm, and spectral clustering algorithm. Preferably, in one embodiment of the present application, all feature vectors are divided into multiple clusters using a k-means clustering algorithm.

[0071] Based on the above analysis, the intra-class deviation value of each feature vector in the mth cluster is calculated:

[0072]

[0073] In the formula, is the intra-class deviation value of the xth eigenvector in the mth cluster, is the number of eigenvectors in the mth cluster, x, y, are the xth eigenvector, yth eigenvector, and feature vectors; , are the xth eigenvector and the xth eigenvector in the mth cluster. The cosine similarity between the yth feature vector and the yth feature vector.

[0074] The greater the difference between text data of the same type of content obtained in the same field, the greater the difference between the corresponding v-dimensional feature vectors in the same cluster. The larger the value of , the more it indicates that even with a certain degree of similarity, there are still certain differences between the feature vectors constructed from the text data under this type of content.

[0075] S003, determining the content complexity of each cluster based on the distribution dispersion of the intra-class deviation value of the feature vector in each cluster; determining the content convertibility between two clusters based on the content complexity combined with the similarity of the feature vectors in the two clusters.

[0076] Specifically, in the tree node The text attribute vector in is formed by text data of the same content. If such content can form more text data under similar semantics, then in the tree node The clusters obtained from all feature vectors of the text attribute vector will also have a certain degree of similarity. For example, in the annual report of a company, turnover, sales and transaction amount have relatively close semantics; in user evaluation text data, "very good", "very good" and "like" may have relatively close semantics.

[0077] Furthermore, if the intra-class deviation values ​​of the feature vectors in two clusters are distributed relatively closely, and are both obtained by dimensionality reduction of text attribute vectors of text data of similar content, then the text data corresponding to the feature vectors in the two clusters may have different expressions in different data sources, but the text data contents have the possibility of being exchanged and combined with each other.

[0078] Therefore, the distribution discreteness of the intra-class deviation value of the feature vector in each cluster is used. Specifically, the distribution variance and range of the intra-class deviation value of the feature vector in each cluster are calculated respectively, and the product of the distribution variance and range is used as the content complexity of each cluster.

[0079] Based on the above analysis, the difference between the content complexity of the two clusters and the similarity between the feature vectors in the two clusters are used to analyze the possibility of the text data content corresponding to the feature vectors of the two clusters being interchangeable and combined, and the content convertibility between the two clusters is determined. Specifically, the calculation formula for the content convertibility between the mth and m+1th clusters is as follows:

[0080]

[0081]

[0082] In the formula, is the similarity feature value between the mth and m+1th clusters, , are the number of feature vectors in the mth and m+1th clusters, respectively. x and h are the xth feature vector in the mth cluster and the hth feature vector in the m+1th cluster, respectively. is the cosine similarity between the xth eigenvector and the hth eigenvector;

[0083] is the content convertibility between the mth and m+1th clusters, , are the content complexity of the mth and m+1th clusters respectively.

[0084] Among them, the higher the similarity between the feature vectors in the mth and m+1th clusters, the higher the similarity between the text attribute vectors corresponding to the feature vectors in the mth and m+1th clusters, and the more likely they are to express similar text content; and the larger the range of intra-class deviation values ​​between the feature vectors in the clusters and the higher the degree of distribution dispersion, the greater the difference between the text data content corresponding to similar text attribute vectors, and the higher the complexity of the text data under such content; and the higher the complexity of the text data corresponding to the feature vectors in the two clusters, the lower the similarity, that is, The larger the value of , the greater the possibility that the words in the text data corresponding to the feature vectors in the two clusters are exchanged and combined into content with similar semantics and appear in different data sources.

[0085] S004, determining the voting weight of the isolated tree based on the content convertibility between the clusters where the feature vectors in the tree nodes are located, and obtaining the abnormality score of each text attribute vector based on the voting weight.

[0086] Furthermore, the tree nodes are calculated separately The content convertibility between all feature vectors obtained by dimensionality reduction of the internal text data is used to construct tree nodes The specific process is as follows:

[0087] First, the tree node Each cluster obtained by clustering all feature vectors obtained by dimensionality reduction of the internal text data is taken as a node, and the content convertibility between two clusters is used as the connection weight between the corresponding two nodes. Tree nodes are obtained based on all clusters obtained by clustering. feature map.

[0088] Next, we need to add the tree nodes The feature map is taken as input and the Prim algorithm is used to generate tree nodes The minimum spanning tree of the feature graph is obtained, wherein the Prim algorithm is a well-known technology and the specific process will not be repeated here.

[0089] It should be noted that the purpose of constructing the minimum spanning tree of tree nodes is that the text attribute vectors in each tree node are formed by text data with the same content, and the minimum spanning tree is a tree node containing All nodes in the feature graph of , and make the tree nodes The processing result of the minimum weight of the connection between all nodes in the feature graph is obtained, which not only The possibility of combinatorial transformation of the text data corresponding to the inner text attribute vector is evaluated, and the cost of combinatorial transformation is also evaluated.

[0090] Based on the above analysis, the connection weights between nodes and the number of nodes in the generated minimum spanning tree are used to evaluate the tree nodes. The degree of distinguishability of the inner text attribute vector.

[0091] Specifically, in the present application, the ratio of the sum of the connection weights between the nodes in the generated minimum spanning tree to the number of nodes is used as the sample segmentation accuracy of the corresponding tree node.

[0092] Among them, in the minimum spanning tree, when the transition between the states of two minimum spanning tree nodes is easier, that is, the greater the connection weight between the two minimum spanning tree nodes, it means that the two minimum spanning tree nodes may be more similar, and the tree nodes in the isolated forest are The lower the difference between different sample states, the higher the accuracy of sample segmentation.

[0093] Furthermore, based on the above steps, the sample segmentation accuracy of each tree node in the ith isolated tree is obtained. At the same time, the tree nodes at the same depth in the ith isolated tree are recorded as the same-layer tree nodes, and the maximum sample segmentation accuracy among the tree nodes at the same layer is used as the segmentation representative value of this layer. The segmentation representative values ​​of each layer are sorted in order from small to large in depth of each layer of the isolated tree, and the sorting result is recorded as the segmentation representative sequence. Based on the above analysis, the voting weight of the ith isolated tree can be obtained. :

[0094]

[0095] in, is the voting weight of the ith isolated tree, , are the bth and b-1th split representative values ​​in the split representative sequence of the ith isolated tree, respectively. ln() is a logarithmic function with a natural constant as the base. is a parameter constant used to prevent the true value from being 0. The size of takes the empirical value of 10, is the depth of the ith isolated tree.

[0096] As the depth of the isolated tree increases, the accuracy of sample segmentation increases, that is, The larger the value, the stronger the sample classification ability of the ith isolated tree is, and the higher the voting weight should be in the isolated forest. The bigger.

[0097] Furthermore, the voting weights of each isolated tree in the isolation forest can be obtained through the above steps. Secondly, the text attribute vector of the text data of each type of content obtained in each field is used as input, and the abnormal score of each text attribute vector is obtained using the isolation forest based on the voting weights of all isolated trees. Among them, the isolation forest algorithm is a well-known technology and will not be described in detail in this embodiment.

[0098] S005, based on the abnormal scores of the text attribute vectors corresponding to all text data in each field and each type of content, the text data that needs to be generalized is screened, and the generalization result is obtained using the generalization model.

[0099] For text data under a category of content in each field, the larger the anomaly score value of the text attribute vector of the text data, the more obvious the difference between the text data under this category of content and the rest of the text data, and the more generalization processing should be performed to reduce the differences that are easy to mine between text data.

[0100] Specifically, for the text data of the a-th category content in any field, the abnormal scores of the text attribute vectors corresponding to all text data of the a-th category content in the field are taken as input, and a data clustering algorithm is adopted. The number of clusters is set to 2, and the output is 2 clusters, and the mean of all abnormal scores in each cluster is calculated, and the text data corresponding to the abnormal scores in the cluster with the largest mean is recorded as a high-difference sample of the a-th category content in the field, and the high-difference sample is taken as an object that needs to be generalized to ensure the security of text data in economic big data. Among them, data clustering is a commonly used technology in the field of data processing, including but not limited to K-means clustering algorithm, DBSCAN clustering algorithm, hierarchical clustering algorithm, ISODATA clustering algorithm. Preferably, in one embodiment of the present application, all abnormal scores are divided into 2 clusters using the K-means clustering algorithm.

[0101] Furthermore, for each high-difference sample of each category of content in each field, the edit distance between each high-difference sample and the high-difference samples of all categories of content in the remaining fields is calculated respectively, and the set of high-difference samples of all categories of content in the remaining fields whose edit distance is less than the distance threshold is used as the anonymous candidate set of each high-difference sample. Preferably, in one embodiment of the present application, the distance threshold takes the empirical value of 3. Secondly, each high-difference sample and the anonymous candidate set of each high-difference sample are used as the input of the k-anonymity algorithm, and the k-anonymity algorithm is used to obtain the generalization processing result of each high-difference sample.

[0102] Among them, the edit distance between texts and the k-anonymity algorithm are commonly used technologies in the field of data generalization, and the specific process will not be repeated here.

[0103] It should be noted that the above sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0104] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0105] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Modifications to the technical solutions recorded in the aforementioned embodiments, or equivalent replacement of some of the technical features therein, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A generalized processing and protection method for text data in economic big data, characterized in that: The method comprises the following steps: Acquire text data of various types of content in different fields from multiple data sources; According to the text data in each category of content in each field, the text attribute vector of each text data is obtained through the PV-DM model, the isolation forest model is trained through the text attribute vector of the text data, each text attribute vector is reduced to a feature vector of a preset dimension by using the PCA principal component analysis method, and the feature vectors obtained by reducing the dimension of all text attribute vectors in each tree node are divided into multiple clusters by using a clustering algorithm, and the intra-class deviation value of each feature vector in the cluster is determined based on the intra-class differences between the feature vectors obtained by reducing the dimension of the text attribute vectors; Determine the content complexity of each cluster based on the distribution discreteness of the intra-class deviation value of the feature vector in each cluster; determine the content convertibility between two clusters based on the content complexity combined with the similarity of the feature vectors in the two clusters; Determine the voting weight of the isolated tree based on the content convertibility between the clusters where the feature vectors in the tree nodes are located, and obtain the abnormality score of each text attribute vector based on the voting weight; Based on the abnormal scores of the text attribute vectors corresponding to all text data in each field and each type of content, the text data that needs to be generalized is screened, and the generalization processing results are obtained using the generalization model.

2. The generalized processing and protection method for text data in economic big data according to claim 1, characterized in that: The training isolation forest model comprises: The text data of each type of content obtained in each field is taken as input, and the PV-DM model is used to obtain the text attribute vector of each text data of each type of content obtained in each field; The length of each text attribute vector is used as the dimension of each sample in the isolation forest algorithm, and the maximum depth of the isolation tree is determined by cross-validation. The text attribute vectors of all text data of each category of content obtained in each field are used as training samples to train the isolation forest model.

3. The generalized processing and protection method for text data in economic big data according to claim 1, characterized in that: The step of determining the intra-class deviation value of each feature vector in the cluster based on the intra-class differences between the feature vectors obtained by dimensionality reduction of the text attribute vectors comprises: Calculate the cosine similarity between any two text attribute vectors in each tree node on each isolated tree, input all the obtained cosine similarities into the Otsu threshold algorithm, use the Otsu threshold algorithm to obtain the segmentation threshold of the similarity, and record the combination of two text attribute vectors with cosine similarity greater than or equal to the segmentation threshold as a true similar combination; Each eigenvector in each cluster is taken as a calculation object, and the absolute value of the difference in cosine similarity between each calculation object, each eigenvector and the same eigenvector is calculated respectively. The mean of the accumulation results of the absolute value of the difference on all eigenvectors in each cluster is taken as the intra-class deviation value of the calculation object.

4. The generalized processing and protection method for text data in economic big data according to claim 1, characterized in that: The determining of the content complexity of each cluster based on the distribution discreteness of the intra-class deviation value of the feature vector in each cluster includes: The distribution variance and range of the intra-class deviation value of the feature vector in each cluster are calculated respectively, and the product of the distribution variance and the range is used as the content complexity of each cluster.

5. The generalized processing and protection method for text data in economic big data according to claim 4 is characterized in that: The determining the content convertibility between the two clusters based on the content complexity combined with the similarity of the feature vectors in the two clusters includes: The mean of the cosine similarity between two feature vectors in two clusters is taken as the similarity feature value between the two clusters. The sum of the content complexity of the two clusters is used as the numerator; The ratio of the molecule to the similarity feature value is taken as the content convertibility between the two clusters.

6. The generalized processing and protection method for text data in economic big data according to claim 1, characterized in that: The method of determining the voting weight of the isolated tree based on the content convertibility between the clusters where the feature vectors in the tree nodes are located, and obtaining the abnormality score of each text attribute vector based on the voting weight, includes: The minimum spanning tree corresponding to each tree node is determined by using the content convertibility between clusters obtained by clustering the feature vectors corresponding to all text attribute vectors in each tree node, and the ratio of the sum of the connection weights between the nodes in the minimum spanning tree to the number of nodes is used as the sample segmentation accuracy of the tree node; The voting weight of each isolated tree is determined based on the sample segmentation accuracy of all tree nodes in each isolated tree; The text attribute vector of the text data of each category of content obtained in each field is taken as input, and the abnormality score of each text attribute vector is obtained using the trained isolation forest model based on the voting weights of all isolated trees.

7. The generalized processing and protection method for text data in economic big data according to claim 6 is characterized in that: The method of determining the minimum spanning tree corresponding to each tree node by using the content convertibility between clusters obtained by clustering the feature vectors corresponding to all text attribute vectors in each tree node comprises: Each cluster obtained by clustering all feature vectors obtained by dimensionality reduction of text data in each tree node is regarded as a node, and the content convertibility between two clusters is used as the connection weight between the corresponding two nodes. Based on all clusters obtained by clustering, the feature graph of each tree node is obtained; The feature map of each tree node is taken as input, and the Prim algorithm is used to generate the minimum spanning tree of the feature map of each tree node.

8. The generalized processing and protection method for text data in economic big data according to claim 7, characterized in that: The step of determining the voting weight of each isolated tree based on the sample segmentation accuracy of all tree nodes in each isolated tree includes: The tree nodes in the same layer of the isolated tree are recorded as the tree nodes of the same layer, and the maximum sample segmentation accuracy among the tree nodes of the same layer is taken as the segmentation representative value of this layer; Sort the segmentation representative values ​​of each layer of the isolation tree from small to large according to the depth of the corresponding layer to obtain a segmentation representative sequence; For each adjacent element in the segmentation representative sequence, calculate the difference between the next element and the previous element in the adjacent elements, and determine the calculation result of the logarithmic function with the sum of the difference and a preset constant as a true number and a natural constant as a base; The average of the accumulated results of the calculation results on the segmentation representative sequence is used as the voting weight of each isolated tree.

9. The generalized processing and protection method for text data in economic big data according to claim 8, characterized in that: The method of screening text data that needs generalization processing based on the abnormal scores of the text attribute vectors corresponding to all text data in each field and each type of content includes: The anomaly scores of the text attribute vectors corresponding to all text data of each category of content in each field are taken as input, and the data clustering algorithm is used. The number of clusters is set to 2, and the output is 2 clusters; The mean of all anomaly scores in each cluster is calculated, and the text data corresponding to the anomaly scores in the cluster with the largest mean is recorded as a high-difference sample of each category of content in each field, and the high-difference sample is taken as an object that needs to be generalized.

10. The generalized processing and protection method for text data in economic big data according to claim 1, characterized in that: The method of obtaining a generalized processing result by using the generalized model includes: For each high-difference sample of each category of content in each field, respectively calculate the edit distance between each high-difference sample and high-difference samples of all categories of content in the remaining fields, and take the set of high-difference samples of all categories of content in the remaining fields whose edit distance is less than the distance threshold as the anonymous candidate set of each high-difference sample; Each high-difference sample and the anonymous candidate set of each high-difference sample are used as inputs of the k-anonymity algorithm, and the k-anonymity algorithm is used to obtain the generalization processing result of each high-difference sample.

Images