Text data homogeneity detection method and device

Abstract

The present invention relates to the technical field of text data detection, and discloses a homogeneity detection method for text data, including: obtaining text data extracted from a business database using a big data platform; inputting the text data into a pre-trained vector conversion model after word segmentation processing to obtain a digitized text feature vector; using a data processing engine as a data calculation and modeling framework, performing similarity calculation on the digitized text feature vector to obtain the similarity between the text feature vectors; extracting a target text feature vector whose similarity is greater than a preset threshold, and using the text data corresponding to the target text feature vector as homogeneous data. The present invention also relates to blockchain technology, and the text data is stored in a blockchain. The present invention can accurately detect homogeneous text data and improve the efficiency of homogeneity detection of text data.

Description

Technical Field

[0001] The present invention relates to the technical field of text data detection, and in particular to a text data homogeneity detection method, device, computer equipment and computer storage medium. Background Art

[0002] With the rise of the Internet era, more and more UGC (User Generated Content) data has been generated. The biggest feature of this type of data is the randomness and repetitiveness (homogenization) of the input content. Users may publish multiple pieces of information with the same content due to input errors of individual characters.

[0003] There are two main methods for detecting homogeneous texts. One is to perform full-text homogeneity detection on the text to be tested. If the homogeneity value of the text data in the text to be tested reaches a homogeneity threshold, it is determined to be homogeneous text data. The other is to extract text data at the same position from the text to be tested as a basis for judgment. If the homogeneity value of the text data at the same position reaches a homogeneity threshold, it is determined to be homogeneous text data.

[0004] The inventors of the present invention discovered during research that for homogeneity detection of massive text data, due to the limitations of microcontroller memory and disk, it is impossible to store and read a large amount of data into the memory for text homogeneity detection. As a result, the number of judgments required by the above two methods will explode exponentially with the increase in the number of texts in the text collection, and the calculation complexity is high, which affects the efficiency of text homogeneity detection and cannot guarantee the accuracy of homogeneity detection. Summary of the invention

[0005] In view of this, the present invention provides a method, apparatus, computer device and computer storage medium for homogeneity detection of text data, the main purpose of which is to solve the problem of low accuracy in homogeneity detection of text data.

[0006] According to one aspect of the present invention, a method for detecting homogeneity of text data is provided, the method comprising:

[0007] Obtain text data extracted from business databases using the big data platform;

[0008] The text data is processed by word segmentation and then input into a pre-trained vector conversion model to obtain a digitized text feature vector;

[0009] Using a data processing engine as a data calculation and modeling framework, similarity calculation is performed on the digitized text feature vectors to obtain the similarity between the text feature vectors;

[0010] The target text feature vector whose similarity is greater than a preset threshold is extracted, and the text data corresponding to the target text feature vector is used as homogenized data.

[0011] In another embodiment of the present invention, the text data is stored in a blockchain, and the text data is segmented and then input into a pre-trained vector conversion model to obtain a digitized text feature vector, specifically including:

[0012] Based on the text features of the text data, the text data is reasonably split to obtain processed text segmentation;

[0013] The processed text segmentation is input into the pre-trained vector conversion model to obtain the digitized text feature vector.

[0014] In another embodiment of the present invention, before the text data is input into a pre-trained vector conversion model after word segmentation to obtain a digitized text feature vector, the method further includes:

[0015] The pre-collected word segmentation sample data is input into the network model for iterative training to obtain a vector conversion model.

[0016] In another embodiment of the present invention, the network model includes a multi-layer structure, and the pre-collected word segmentation sample data is input into the network model for iterative training to obtain a vector conversion model, which specifically includes:

[0017] Using the first layer structure of the network model, a central word is selected from the word segmentation set after the word segmentation processing of the text data;

[0018] Using the second layer structure of the network model, each segmented word and a pre-selected central word are combined into a prediction pair according to a specified window size, and a weight parameter between each segmented word and the central word is predicted;

[0019] The third layer structure of the network model is used to adjust the weight parameter between each segment word and the central word to obtain a digitized text feature vector.

[0020] In another embodiment of the present invention, the data processing engine is used as a data calculation and modeling framework to perform similarity calculation on the digitized text feature vectors to obtain the similarity between the text feature vectors, specifically including:

[0021] Use the data processing engine as the data calculation and modeling framework, and select local sensitive models from the preset algorithm library;

[0022] Mapping the digitized text feature vector to a random unit vector using the local sensitivity model to form a mapping result;

[0023] The text feature vectors are allocated to hash buckets based on the mapping results, so that the similarities between the text feature vectors in the hash buckets are within a preset range.

[0024] In another embodiment of the present invention, the local sensitivity model is used to map the digitized text feature vector to a random unit vector to form a mapping result, which specifically includes:

[0025] Calculating the Euclidean metric of the text feature vector using the local sensitivity model;

[0026] The Euclidean metric of the text feature vector is mapped to a random unit vector to form a mapping result.

[0027] In another embodiment of the present invention, after extracting the target text feature vector whose similarity is greater than a preset threshold and using the text data corresponding to the target text feature vector as homogenized data, the method further includes:

[0028] The homogenized data is sent to a big data storage layer, so that the big data storage layer performs similarity marking or query on the homogenized data.

[0029] According to another aspect of the present invention, a text data homogeneity detection device is provided, the device comprising:

[0030] An acquisition unit, used to acquire text data extracted from a business database using a big data platform;

[0031] A processing unit, used to perform word segmentation on the text data and then input the text data into a pre-trained vector conversion model to obtain a digitized text feature vector;

[0032] A calculation unit, used to use a data processing engine as a data calculation and modeling framework to perform similarity calculation on the digitized text feature vectors to obtain the similarity between the text feature vectors;

[0033] The extraction unit is used to extract the target text feature vector whose similarity is greater than a preset threshold, and use the text data corresponding to the target text feature vector as homogenized data.

[0034] In another embodiment of the present invention, the text data is stored in a blockchain, and the processing unit includes:

[0035] A splitting module, used to reasonably split the text data based on the text features of the text data to obtain processed text segmentation;

[0036] The conversion module is used to input the processed text segmentation into a pre-trained vector conversion model to obtain the digitized text feature vector.

[0037] In another embodiment of the present invention, the device further comprises:

[0038] The training unit is used to input the pre-collected word segmentation sample data into the network model for iterative training before inputting the pre-trained vector conversion model after word segmentation to obtain the digitized text feature vector to obtain the vector conversion model.

[0039] In another embodiment of the present invention, the network model includes a multi-layer structure, and the training unit includes:

[0040] A selection module, used for selecting a central word from a segmentation set after the segmentation processing of the text data by using the first layer structure of the network model;

[0041] A prediction module, used to use the second layer structure of the network model to form a prediction pair with each segmented word and a pre-selected central word according to a specified window size, and predict a weight parameter between each segmented word and the central word;

[0042] The adjustment module is used to adjust the weight parameters between each segment word and the central word by using the third layer structure of the network model to obtain a digitized text feature vector.

[0043] In another embodiment of the present invention, the computing unit comprises:

[0044] A selection module is used to select a local sensitive model from a preset algorithm library using a data processing engine as a data calculation and modeling framework;

[0045] A mapping module, used to map the digitized text feature vector to a random unit vector using the local sensitivity model to form a mapping result;

[0046] An allocating module is used to allocate the text feature vectors to hash buckets based on the mapping results, so that the similarities between the text feature vectors in the hash buckets are within a preset range.

[0047] In another embodiment of the present invention, the mapping module includes:

[0048] A calculation submodule, used for calculating the Euclidean metric of the text feature vector using the local sensitivity model;

[0049] The mapping submodule is used to map the Euclidean metric of the text feature vector to a random unit vector to form a mapping result.

[0050] In another embodiment of the present invention, the device further comprises:

[0051] A sending unit is used to send the homogenized data to a storage layer of big data after extracting the target text feature vector whose similarity is greater than a preset threshold and taking the text data corresponding to the target text feature vector as homogenized data, so that the storage layer of big data performs similarity marking or query on the homogenized data.

[0052] According to another aspect of the present invention, there is provided a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a method for detecting homogeneity of text data when executing the computer program.

[0053] According to another aspect of the present invention, there is provided a computer storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for detecting homogeneity of text data are implemented.

[0054] By means of the above technical scheme, the present invention provides a method and device for detecting homogeneity of text data, which obtains text data extracted from a business database using a big data platform, and inputs the text data into a pre-trained vector conversion model after word segmentation to obtain a digitized text feature vector, uses a data processing engine as a data calculation and modeling framework, performs similarity calculation on the digitized text feature vector, obtains the similarity between the text feature vectors, extracts a target text feature vector whose similarity is greater than a preset threshold, and uses the text data corresponding to the target text feature vector as homogeneous data. Compared with the method for homogeneity detection of text data at the same position in the prior art, the present application makes full use of the characteristics of various components of big data, and constructs an algorithm for detecting text homogeneity based on a vector conversion model and a local sensitive algorithm. The algorithm can accurately detect homogeneous text data under given threshold conditions, thereby improving the efficiency of homogeneity detection of text data. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0056] Figure 1 A schematic diagram of a process flow of a text data homogeneity detection method provided by an embodiment of the present invention is shown;

[0057] Figure 2 A schematic flow chart of another method for detecting homogeneity of text data provided by an embodiment of the present invention is shown;

[0058] Figure 3An overall framework diagram of performing homogeneity detection on text data provided by an embodiment of the present invention is shown;

[0059] Figure 4 A schematic diagram showing the structure of a text data homogeneity detection device provided by an embodiment of the present invention is shown;

[0060] Figure 5 A schematic structural diagram of another text data homogeneity detection device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0061] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0062] The embodiment of the present invention provides a method for detecting homogeneity of text data, which can accurately detect homogeneous text data and improve the efficiency of homogeneity detection of text data. Figure 1 As shown, the method includes:

[0063] 101. Obtain text data extracted from the business database using the big data platform.

[0064] Among them, the big data platform can use Hive as the underlying storage. The text data here includes the background data extracted from the business database. The background data is usually UGC (User Generated Content) text data, thereby ensuring the consistency of the UGC text data under the big data platform and the UGC text data in the business database.

[0065] Specifically, in the process of extracting text data from the business database, text processing technology can be used to specify the type of entity, relationship, event and other factual information from the natural language text, and form structured data output. While using text processing technology, it is necessary to consider the rule constraints of text data, for example, the extraction of a certain type of text data, the extraction of a certain structured text data. It should be emphasized that in order to further ensure the privacy and security of the above text data, the above text data can also be stored in a blockchain node.

[0066] 102. After word segmentation, the text data is input into a pre-trained vector conversion model to obtain a digitized text feature vector.

[0067] Normally, text data is usually a whole paragraph of text. Specifically, the word segmentation tool can be used to segment the text data and reasonably split the whole paragraph of text according to the part of speech. For example, the Hanlp toolkit can be used to split the text data into words. At the same time, single words or commonly used modal particles, pause words and conjunctions are removed to retain the words with real meaning as much as possible.

[0068] Among them, the pre-trained vector conversion model can use the neural network algorithm to establish a network model with a 3-layer neural network structure, and train the network for the word set of each text in the expected sample for the pre-collected corpus sample. When the network training is completed, each row of data in the feature matrix of the hidden layer corresponds to the numerical feature of each word segmentation, thereby realizing the numerical segmentation.

[0069] 103. Using the data processing engine as a data calculation and modeling framework, similarity calculation is performed on the digitized text feature vectors to obtain the similarity between the text feature vectors.

[0070] Since the Spark platform has strong data processing capabilities and fast computing speed, the data processing engine here can use the Spark platform to process massive digitized text vectors. Specifically, you can select the Bucketed Random Projection LSH local sensitivity model in the Spark ML algorithm library. This local sensitivity model can map text feature vectors to random unit vectors v respectively, and divide the mapping results into hash buckets. Similar text feature vectors will be divided into the same bucket, and then the text feature vectors in the same bucket will be calculated and compared in parallel.

[0071] 104. Extract the target text feature vector whose similarity is greater than a preset threshold, and use the text data corresponding to the target text feature vector as homogenized data.

[0072] In the embodiment of the present invention, the target text vectors with high similarity can be stored in the result table, which mainly records the text number, similar text number, text similarity, text content and similar text content, etc. In order to facilitate subsequent queries, the result table can also be written to the hbase component at the same time to provide millisecond-level fast query capabilities.

[0073] The embodiment of the present invention provides a method for detecting homogeneity of text data, which obtains text data extracted from a business database using a big data platform, performs word segmentation on the text data and then inputs it into a pre-trained vector conversion model to obtain a digitized text feature vector, uses a data processing engine as a data calculation and modeling framework, performs similarity calculation on the digitized text feature vector, obtains the similarity between the text feature vectors, extracts a target text feature vector whose similarity is greater than a preset threshold, and uses the text data corresponding to the target text feature vector as homogeneous data. Compared with the method for detecting homogeneity of text data at the same position in the prior art, the present application makes full use of the characteristics of various components of big data, and constructs an algorithm for detecting text homogeneity based on a vector conversion model and a local sensitive algorithm. The algorithm can accurately detect homogeneous text data under given threshold conditions, thereby improving the efficiency of homogeneity detection of text data.

[0074] The embodiment of the present invention provides another method for detecting homogeneity of text data, which can accurately detect homogeneous text data and improve the efficiency of homogeneity detection of text data. Figure 2 As shown, the method includes:

[0075] 201. Obtain text data extracted from a business database using a big data platform.

[0076] In the embodiment of the present invention, since the big data platform can horizontally expand the storage capacity and computing power of the cluster, it can solve the problem of insufficient processing power of a traditional single machine. The big data platform here can theoretically process an unlimited amount of data.

[0077] 202. Use the pre-collected word segmentation sample data to input into the network model for iterative training to obtain a vector conversion model.

[0078] Among them, the network model includes a multi-layer structure, which is an input layer, a hidden layer and an output layer. Specifically, the first layer of the network model can be used to select a central word from a segmentation set after the text data is segmented, and the second layer of the network model can be used to form a prediction pair with each segmentation and the pre-selected central word according to a specified window size, and the weight parameter between each segmentation and the central word is predicted, and the third layer of the network model can be used to adjust the weight parameter between each segmentation and the central word to obtain a digitized text feature vector.

[0079] In specific application scenarios, a window of appropriate size can be taken as the context. The input layer reads the word segmentations in the window, and the vectors formed by the word segmentations (K dimensions, initially random) are added together to form K nodes in the hidden layer. The output layer is a huge binary tree. The leaf nodes represent all the word segmentations in the corpus (the corpus contains V independent words, then the binary tree has |V| leaf nodes), and the algorithm used to construct the entire binary tree is the Huffman tree. In this way, for each word segmentation in the leaf node, there will be a globally unique code in the form of "010011". It is worth remembering that the left subtree is 1 and the right subtree is 0. Next, each node in the hidden layer will have an edge with the inner node of the binary tree. There will be K edges for each inner node of the binary tree, and each edge will also have a weight.

[0080] In the process of determining the specific weight parameters, for a word w_t in the corpus, it corresponds to a leaf node of the binary tree, so there must be a binary code, such as "010011". In the training stage, when a context is given and the subsequent word w_t is to be predicted, it can be traversed from the root node of the binary tree. The goal here is to predict each bit of the binary number of this word. That is, for a given context, the binary coding probability of the predicted word is maximized, so that at the root node, the probability of the word vector and the bit=1 obtained by logistic calculation connected to the root node is as close to 0 as possible. In the second layer, the probability of bit=1 is as close to 1 as possible. Repeat the execution to multiply the probabilities calculated above, that is, the probability P(w_t) of the target word w_t under the current network model is obtained. The residual for the current sample is 1-P(w_t). The gradient descent method can be used to train this network model to obtain all the weight parameters.

[0081] 203. Based on the text features of the text data, the text data is reasonably split to obtain processed text segmentations.

[0082] In the embodiment of the present invention, a word segmentation tool may be used to reasonably split the text data, for example, jieba word segmentation, hanLP, etc. A bag-of-words model may also be used to reasonably split the text data, which is not limited here.

[0083] 204. Input the processed text segmentation into a pre-trained vector conversion model to obtain a digitized text feature vector.

[0084] In an embodiment of the present invention, the spark ML algorithm library is used in the training process of the network model, and the calculation results obtained by the Word2Vec model used are selected in the spark ML algorithm library. Specifically, in the process of using the Word2Vec model to train the vector conversion model, since the Word2Vec model itself is a shallow neural network model with a multi-level structure of a neural network, the input of the Word2Vec model is a word segmentation set corresponding to the text data, and the text data is composed of multiple sentences. For each sentence, it is a word segmentation set of a sentence. Further, the Word2Vec model will construct a prediction pair according to the specified window, and use the previous word segmentation to predict the relevance of the following word, so that the word segmentation relationship in all sentences can be obtained. After continuous iterative training through each layer of the network, the parameters of the hidden layer can be obtained. The parameters of the hidden layer are equivalent to the word vectors corresponding to the word segmentation in the text data, and then the word vectors corresponding to the word segmentation in the text data are added to obtain the sentence vector. For example, for the sentence "I love Beijing", the input of the Word2Vec model is the word segmentation set of the sentence: [I, love, Beijing], and the output of the Word2Vec model is the text feature vector after the sentence is digitized: [0.1, 0.5, 0.65, 0.25].

[0085] 205. Use the data processing engine as the data calculation and modeling framework, and select a local sensitive model from the preset algorithm library.

[0086] 206. Map the digitized text feature vector to a random unit vector using the local sensitivity model to form a mapping result.

[0087] In the embodiment of the present invention, the local sensitivity model can be used to calculate the Euclidean metric of the text feature vector, and the Euclidean metric of the text feature vector can be mapped to a random unit vector to form a mapping result.

[0088] 207 . Distribute the text feature vectors into hash buckets based on the mapping results, so that the similarities between the text feature vectors in the hash buckets are within a preset range.

[0089] Specifically, in the process of similarity calculation of the digitized text feature vector, the local sensitive model uses the random projection formula of the Euclidean metric to map the text feature vector to a random unit vector v. The random bucket projection is the LSH family for the Euclidean distance. The definition of the Euclidean metric is as follows:

[0090]

[0091] Among them, LSH family maps the feature vector of vector X to a random unit vector V and divides the mapping results into hash buckets;

[0092]

[0093] Where r is the user-defined bucket length, which can be used to control the average size of the hash buckets (and therefore the number of buckets). Larger bucket lengths (i.e., fewer buckets) increase the probability that features hash to the same bucket (increasing the number of true and false positives). Random Pass Projection accepts arbitrary vectors as input features and supports both sparse and dense vectors.

[0094] 208. Extract the target text feature vector whose similarity is greater than a preset threshold, and use the text data corresponding to the target text feature vector as homogenized data.

[0095] 209. Send the homogenized data to a big data storage layer, so that the big data storage layer performs similarity marking or query on the homogenized data.

[0096] In the embodiment of the present invention, the overall framework diagram for performing homogeneity detection on text data can be as follows: Figure 3 As shown in the figure, it includes the storage layer of basic data, the text data processing layer, the similarity calculation layer and the result output layer. The layered idea is used to decouple the processing logic of each layer, and the data in each layer can be flexibly expanded.

[0097] Specifically, the storage layer of basic data uses Hive as the underlying storage. The data source includes the background data extracted from the business database. Simple data extraction is performed on the text data to ensure the consistency of the UGC text data under the big data platform and the UGC text data in the business database; the text data processing layer mainly performs word segmentation on the user's UGC text data and digitizes the data after word segmentation. Since text data cannot be directly involved in calculations, it is necessary to split the text data according to the characteristics of word segmentation. A whole paragraph of text is reasonably split according to the part of speech. This algorithm uses the HanLp toolkit to split the text into words. At the same time, single words or commonly used modal particles, pause words and conjunctions are removed to retain words with real meaning as much as possible. Using the neural network algorithm, a three-layer neural network structure is established, all words are regarded as a corpus, and the network is trained for the word set of each text. After the network training is completed, each row of data in the feature matrix of the hidden layer corresponds to the numerical feature vector of each word, thus realizing the numericalization of words. The sparkML algorithm library is used for model training, and the applicable Word2Vec model is selected in the spark ML library to obtain the calculation results; the similarity calculation layer mainly calculates the similarity of the numerical text. Spark is used as the data calculation and modeling framework to process massive text vectors. The Bucketed Random Projection LSH local sensitive model is selected in the spark ML library to map the text feature vectors to random unit vectors v respectively, and the mapping results are divided into hash buckets. Therefore, similar texts will be divided into the same bucket, and then the data in the same bucket are calculated in parallel to compare the similarity, which significantly improves the calculation efficiency; the result output layer mainly stores highly similar text data, and similar pairs are marked for similar data. The result table mainly records the text number, similar text number, text similarity, text content and similar text content, and writes the results to the hbase component at the same time, providing millisecond-level fast query capabilities.

[0098] It is understandable that the underlying storage uses Hive instead of traditional RDBMS such as Mysql, which can store larger amounts of data. The computing layer uses Spark to process larger amounts of data and has faster computing speeds. The Word2Vec algorithm based on neural networks is used to vectorize the text, making full use of the words in the entire text library, while considering the impact of the order of words on the text, making the vectorization results of a single word more accurate. At the same time, the BucketedRandomProjection LSH algorithm is also used in the model to calculate the similarity of the feature vector of the text based on the Euclidean distance, and the results after bucketing can be quickly calculated in parallel, greatly improving the computing efficiency. The data results are sorted to display the data results in a more user-friendly way. The result output layer storage uses Hbase components, which provides millisecond-level query of similar results of specified texts under massive data. From the perspective of architecture, each block is modularized, and the function of each module can be upgraded separately, thereby improving the upgrade capability of the entire algorithm.

[0099] Further, as Figure 1 In a specific implementation of the method, the present invention provides a text data homogeneity detection device, such as Figure 4 As shown, the device includes: an acquisition unit 31, a processing unit 32, a calculation unit 33, and an extraction unit 34.

[0100] The acquisition unit 31 may be used to acquire text data extracted from a business database using a big data platform;

[0101] The processing unit 32 may be used to perform word segmentation on the text data and then input the text data into a pre-trained vector conversion model to obtain a digitized text feature vector;

[0102] The calculation unit 33 can be used to use the data processing engine as a data calculation and modeling framework to perform similarity calculation on the digitized text feature vectors to obtain the similarity between the text feature vectors;

[0103] The extraction unit 34 may be used to extract the target text feature vectors whose similarity is greater than a preset threshold, and use the text data corresponding to the target text feature vectors as homogenized data.

[0104] The embodiment of the present invention provides a homogeneity detection device for text data, which obtains text data extracted from a business database using a big data platform, and inputs the text data into a pre-trained vector conversion model after word segmentation to obtain a digitized text feature vector, uses a data processing engine as a data calculation and modeling framework, performs similarity calculation on the digitized text feature vector, obtains the similarity between the text feature vectors, extracts a target text feature vector whose similarity is greater than a preset threshold, and uses the text data corresponding to the target text feature vector as homogeneous data. Compared with the method of performing homogeneity detection on text data at the same position in the prior art, the present application makes full use of the characteristics of various components of big data, and constructs an algorithm for detecting text homogeneity based on a vector conversion model and a local sensitive algorithm. The algorithm can accurately detect homogeneous text data under a given threshold condition, thereby improving the efficiency of homogeneity detection on text data.

[0105] As Figure 4 Further description of the homogeneity detection device for text data shown in, Figure 5 is a schematic diagram of the structure of another text data homogeneity detection device according to an embodiment of the present invention. Figure 5 As shown, the processing unit 32 includes:

[0106] The splitting module 321 can be used to reasonably split the text data based on the text features of the text data to obtain processed text segmentation;

[0107] The conversion module 322 can be used to input the processed text segmentation into a pre-trained vector conversion model to obtain a digitized text feature vector. It should be emphasized that in order to further ensure the privacy and security of the above text data, the above text data can also be stored in a node of a blockchain.

[0108] In a specific application scenario, the device further includes:

[0109] The training unit 35 is used to input the pre-collected word segmentation sample data into the network model for iterative training to obtain the vector conversion model before inputting the pre-trained vector conversion model after word segmentation to obtain the digitized text feature vector.

[0110] In a specific application scenario, the network model includes a multi-layer structure, and the training unit 35 includes:

[0111] A selection module 351 can be used to select a central word from a segmented word set after the text data segmentation process using the first layer structure of the network model;

[0112] The prediction module 352 may be used to use the second layer structure of the network model to form a prediction pair with each segmented word and a pre-selected central word according to a specified window size, and predict a weight parameter between each segmented word and the central word;

[0113] The adjustment module 353 can be used to adjust the weight parameter between each segment word and the central word by using the third layer structure of the network model to obtain a digitized text feature vector.

[0114] In a specific application scenario, the calculation unit 33 includes:

[0115] The selection module 331 can be used to select a local sensitive model from a preset algorithm library using a data processing engine as a data calculation and modeling framework;

[0116] A mapping module 332 may be used to map the digitized text feature vector to a random unit vector using the local sensitivity model to form a mapping result;

[0117] The allocation module 333 may be configured to allocate the text feature vectors to hash buckets based on the mapping result, so that the similarities between the text feature vectors in the hash buckets are within a preset range.

[0118] In a specific application scenario, the mapping module 332 includes:

[0119] A calculation submodule 3321 may be used to calculate the Euclidean metric of the text feature vector using the local sensitivity model;

[0120] The mapping submodule 3322 may be used to map the Euclidean metric of the text feature vector to a random unit vector to form a mapping result.

[0121] In a specific application scenario, the device further includes:

[0122] The sending unit 36 ​​can be used to send the homogenized data to the storage layer of big data after extracting the target text feature vector whose similarity is greater than a preset threshold and taking the text data corresponding to the target text feature vector as homogenized data, so that the storage layer of big data performs similarity marking or query on the homogenized data.

[0123] It should be noted that for other corresponding descriptions of the functional units involved in the text data homogeneity detection device provided in this embodiment, reference can be made to Figure 1 , Figure 2 The corresponding description in will not be repeated here.

[0124] Based on the above Figure 1 , Figure 2The method shown in the embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned Figure 1 , Figure 2 The homogeneity detection method for text data is shown.

[0125] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0126] Based on the above Figure 1 , Figure 2 The method shown, and Figure 4 , Figure 5 In order to achieve the above-mentioned purpose, the embodiment of the present application also provides a computer device, which can be a personal computer, a server, a network device, etc. The physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 1 , Figure 2 The homogeneity detection method for text data is shown.

[0127] Optionally, the computer device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.

[0128] Those skilled in the art will appreciate that the physical device structure of the text data homogeneity detection apparatus provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0129] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the above-mentioned computer device, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the physical device.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform, or by hardware. By applying the technical solution of the present application, compared with the current prior art, the present application makes full use of the characteristics of various components of big data, and constructs an algorithm for detecting text homogeneity based on a vector conversion model and a local sensitivity algorithm. Under given threshold conditions, the algorithm can accurately detect homogeneous text data and improve the efficiency of homogeneity detection of text data.

[0131] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0132] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A method for detecting homogeneity of text data, characterized in that: The method comprises: Obtain text data extracted from business databases using the big data platform; The text data is processed by word segmentation and then input into a pre-trained vector conversion model to obtain a digitized text feature vector; The method uses a data processing engine as a data calculation and modeling framework to perform similarity calculation on the digitized text feature vectors to obtain the similarity between the text feature vectors, specifically comprising: using the data processing engine as a data calculation and modeling framework, selecting a local sensitivity model from a preset algorithm library; using the local sensitivity model to map the digitized text feature vectors to a random unit vector to form a mapping result; and allocating the text feature vectors to a hash bucket based on the mapping result, so that the similarity between the text feature vectors in the hash bucket is within a preset range; Extract the target text feature vector whose similarity is greater than a preset threshold, and use the text data corresponding to the target text feature vector as homogenized data; Before the text data is input into a pre-trained vector conversion model after word segmentation to obtain a digitized text feature vector, pre-collected word segmentation sample data is input into a network model for iterative training to obtain a vector conversion model. The network model includes a multi-layer structure, specifically including: using the first layer structure of the network model to select a central word from a set of word segmentations after the text data is segmented; using the second layer structure of the network model to form a prediction pair with each word segmentation and the pre-selected central word according to a specified window size, and predicting a weight parameter between each word segmentation and the central word; using the third layer structure of the network model to adjust the weight parameter between each word segmentation and the central word to obtain a digitized text feature vector.

2. The method according to claim 1, characterized in that The text data is stored in the blockchain, and the text data is segmented and then input into a pre-trained vector conversion model to obtain a digitized text feature vector, specifically including: Based on the text features of the text data, the text data is reasonably split to obtain processed text segmentation; The processed text segmentation is input into the pre-trained vector conversion model to obtain the digitized text feature vector.

3. The method according to claim 1, characterized in that The local sensitivity model is used to map the digitized text feature vector to a random unit vector to form a mapping result, which specifically includes: Calculating the Euclidean metric of the text feature vector using the local sensitivity model; The Euclidean metric of the text feature vector is mapped to a random unit vector to form a mapping result.

4. The method according to any one of claims 1 to 3, characterized in that After extracting the target text feature vector whose similarity is greater than a preset threshold and using the text data corresponding to the target text feature vector as homogenized data, the method further includes: The homogenized data is sent to a big data storage layer, so that the big data storage layer performs similarity marking or query on the homogenized data.

5. A homogeneity detection device for text data, characterized in that: The device comprises: An acquisition unit, used to acquire text data extracted from a business database using a big data platform; A processing unit, used to perform word segmentation on the text data and then input the text data into a pre-trained vector conversion model to obtain a digitized text feature vector; A calculation unit, used to use a data processing engine as a data calculation and modeling framework to perform similarity calculation on the digitized text feature vectors to obtain the similarity between the text feature vectors; An extraction unit, used for extracting the target text feature vector whose similarity is greater than a preset threshold, and using the text data corresponding to the target text feature vector as homogenized data; The device further comprises: a training unit, which is used to input the pre-collected word segmentation sample data into the network model for iterative training to obtain the vector conversion model before inputting the pre-trained vector conversion model into the pre-trained vector conversion model after the word segmentation processing of the text data to obtain the digitized text feature vector; The network model includes a multi-layer structure, and the training unit includes: a selection module, which is used to select a central word from a segmentation set after the segmentation processing of the text data using the first layer structure of the network model; a prediction module, which is used to form a prediction pair with each segmentation and a pre-selected central word according to a specified window size using the second layer structure of the network model, and predict the weight parameter between each segmentation and the central word; an adjustment module, which is used to adjust the weight parameter between each segmentation and the central word using the third layer structure of the network model to obtain a digitized text feature vector; The calculation unit includes: a selection module, which is used to use the data processing engine as a data calculation and modeling framework to select a local sensitivity model from a preset algorithm library; a mapping module, which is used to map the digitized text feature vector to a random unit vector using the local sensitivity model to form a mapping result; and an allocation module, which is used to allocate the text feature vector to a hash bucket based on the mapping result so that the similarity between the text feature vectors in the hash bucket is within a preset range.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images