Dynamic document deduplication method and system oriented to big data environment

By preprocessing and natural language processing of original text data, calculating TF-IDF values ​​and generating hash features, combined with a distributed computing framework, efficient dynamic document deduplication in large-scale data environments is achieved, and the existing technology is solved Problems of low computing efficiency and high misjudgment rate.

CN118897947APending Publication Date: 2024-11-05GUIZHOU POWER GRID CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410846433.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-27
Publication Date
2024-11-05

AI Technical Summary

Technical Problem

The existing dynamic document deduplication method has low computational efficiency, slow processing speed and high misjudgment rate in large-scale data environments, making it difficult to achieve efficient and accurate deduplication.

Method used

The original text data is collected for preprocessing, keyword segmentation is performed based on natural language processing methods, TF-IDF values ​​are calculated, keyword feature sets and identifiers are generated through hash functions, and distributed computing frameworks are constructed for parallel deduplication processing.

Benefits of technology

It improves the accuracy and efficiency of data processing, reduces the misjudgment rate, realizes efficient deduplication in large-scale data environments, and solves the efficiency bottleneck problem in the existing technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897947A_ABST
    Figure CN118897947A_ABST
Patent Text Reader

Abstract

The invention discloses a dynamic document deduplication method and system oriented to a big data environment, and relates to the technical field of dynamic document deduplication, and the method comprises the following steps: collecting and preprocessing original text data, and carrying out keyword segmentation on the preprocessed text data based on a natural language processing method; calculating a TF-IDF value of each keyword in the segmented text data to obtain frequency text data, generating a keyword feature set based on a hash function, and generating an identifier; and comparing the similarities of the documents through the identifiers, and constructing a distributed computing framework to perform parallel de-duplication processing on the comparison process. According to the method, keyword segmentation is performed on the preprocessed text data based on a natural language processing method, the structured level of the data is improved, the efficiency and accuracy of data processing are improved through TF-IDF value calculation and Hash function mapping, and the efficiency and accuracy of data processing are improved through parallel calculation and task distribution. The problem of efficiency bottleneck existing in large-scale data processing in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic document deduplication, and in particular to a dynamic document deduplication method and system for a big data environment. Background Art

[0002] With the rapid development and widespread application of big data technology, the demand for data processing and analysis in all walks of life is increasing. Especially in the fields of Internet, social media, e-commerce, etc., a large amount of unstructured text data is generated and stored. These text data contain rich user behavior information, market trends and business opportunities, but also bring problems of data redundancy and duplication, which has a serious impact on the efficiency of data storage, processing and analysis. In order to improve the efficiency of data utilization, dynamic document deduplication technology came into being.

[0003] Although the existing dynamic document deduplication technology has solved the problem of data redundancy to a certain extent, it still has many shortcomings when processing large-scale data. Traditional string matching methods cannot effectively capture semantic information when facing unstructured text data, and are prone to misjudgment, resulting in unsatisfactory deduplication effects. Although the NLP-based deduplication method can identify keywords in the text and perform semantic analysis, in the big data environment, the amount of calculation is huge and the processing speed is slow, making it difficult to meet the needs of real-time processing. In addition, although the existing distributed computing methods have improved the efficiency of data processing, in actual applications, problems such as uneven data distribution and unbalanced load still exist, affecting the overall processing performance. Summary of the invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: the existing dynamic document deduplication method has the problems of low computational efficiency, slow processing speed, high misjudgment rate and how to achieve efficient and accurate deduplication in a large-scale data environment.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a dynamic document deduplication method for a big data environment, comprising collecting and preprocessing original text data, performing keyword segmentation on the preprocessed text data based on a natural language processing method; calculating the TF-IDF value of each keyword in the segmented text data to obtain frequency text data, generating a keyword feature set and an identifier based on a hash function; comparing the similarity of each document through the identifier, and constructing a distributed computing framework to perform parallel deduplication processing on the comparison process.

[0007] As a preferred solution of the dynamic document deduplication method for a big data environment described in the present invention, the preprocessing includes collecting original text data, cleaning and converting the original text data; the source of the original text data includes articles, news, blogs, and social media; cleaning includes removing special characters, punctuation marks, numbers, and stop words; conversion includes converting the original text data into lowercase letters.

[0008] As a preferred solution of the dynamic document deduplication method for a big data environment described in the present invention, the keyword segmentation includes performing word segmentation processing on the preprocessed text data based on a natural language processing method, obtaining a word vector sequence, inputting the word vector sequence into a context encoder to obtain a word granularity context semantic feature vector, inputting the word vector sequence into a bidirectional long short-term memory neural network model to obtain a segment granularity context semantic feature vector, fusing the word granularity context semantic feature vector and the segment granularity context semantic feature vector to obtain a multi-granularity fusion feature vector, which is expressed as:

[0009] V a =αV b +βV c

[0010] Among them, V a represents the multi-granularity fusion feature vector, α represents the weighting parameter of the word-granularity contextual semantic feature vector, β represents the weighting parameter of the segment-granularity contextual semantic feature vector, V b Represents the word-level contextual semantic feature vector, V c Representing the segment-granularity context semantic feature vector; constructing a corpus, generating a corpus feature vector of the preprocessed text data, obtaining the erroneous words and the position subscripts of the erroneous words based on the feature differences between the multi-granularity fusion feature vector and the corpus feature vector, and correcting the preprocessed text data through the position subscripts of the erroneous words.

[0011] As a preferred solution of the dynamic document deduplication method for a big data environment described in the present invention, wherein: the obtaining of frequency text data includes obtaining segmented text data by keyword segmentation, and calculating the TF-IDF value of each keyword in the segmented text data, which is expressed as:

[0012] TF-IDF(t,d,D)=TF(t,d)×IDF(t,D)

[0013] Where t represents a keyword, d represents segmented text data, D represents a data set, TF-IDF(t,d,D) represents the TF-IDF value of keyword t in segmented text data d, TF(t,d) represents the frequency of keyword t in segmented text data d, IDF(t,D) represents the inverse document frequency of keyword t in data set D, and the calculation of inverse document frequency is expressed as:

[0014]

[0015] Among them, |D| represents the total number of segmented text data d in the dataset, and |{d∈D:t∈d| represents the number of segmented text data d containing keyword t.

[0016] As a preferred solution of the dynamic document deduplication method for a big data environment described in the present invention, wherein: the generating identifier includes storing the keywords and the corresponding frequencies in a hash table according to the frequency of occurrence of each keyword in the frequency text data, and sorting them from high to low according to the frequency of occurrence, setting a frequency threshold θ, and selecting keywords whose frequency of occurrence is greater than or equal to the frequency threshold, which is expressed as:

[0017] S={t∈T|TF(t,d)≥θ}

[0018] Among them, S represents the selected keyword set, and the feature information of the selected keywords is generated by the hash function to obtain the keyword feature set, which is expressed as:

[0019]

[0020] Among them, F is the keyword feature matrix, TF-IDF(t m ,d n ,D) indicates the keyword t m In segmenting text data n TF-IDF value in, m represents the number of keywords in the feature matrix, n represents the number of documents in the feature matrix, and the key features are mapped by hash functions. Each hash function maps the feature matrix to a hash value, which is expressed as:

[0021]

[0022] Among them, h k (F) is represented by the output of the kth hash function, a' ij represents the random constant that weights the elements in the i-th row and j-th column of the feature matrix, b' k represents the offset constant of the kth hash function, mod represents the modulus operation, p k Represents the prime number used by the kth hash function, normalizes the result of the hash function, and intercepts the binary bits of a fixed length to generate the final identifier, which is expressed as:

[0023]

[0024] Here, H(F) represents the final identifier, K represents the number of hash functions used, and |.| L It means taking binary bits of length L.

[0025] As a preferred solution of the dynamic document deduplication method for a big data environment described in the present invention, wherein: the similarity comparison of each document includes assigning identifiers to different computing nodes for parallel processing, and each computing node is responsible for calculating the similarity between the assigned identifier pairs, which is expressed as:

[0026]

[0027] Among them, X i'j' Represents the similarity between the i'th document and the j'th document, H(F) i' represents the final identifier of the i'th document, H(F) j' represents the final identifier of the j'th document, H(F) i'k Indicates the kth bit of the final identifier of the i'th document, H(F) j'k Represents the kth bit of the final identifier of the j'th document.

[0028] As a preferred solution of the dynamic document deduplication method for a big data environment described in the present invention, the parallel deduplication processing includes building a distributed computing framework to perform parallel deduplication processing on the comparison process, and each computing node performs deduplication judgment on the responsible similarity matrix according to a preset similarity threshold, which is expressed as:

[0029]

[0030] Among them, R i'j' represents the deduplication result between the i'th document and the j'th document, τ represents the similarity threshold, when the similarity of two documents is X i'j' When the similarity X between two documents is greater than or equal to the threshold τ, the two documents are duplicates and need to be deduplicated. i'j' When it is less than the threshold τ, the two documents are not duplicates and do not need to be deduplicated, so the documents are retained.

[0031] Another object of the present invention is to provide a dynamic document deduplication system for a big data environment, which can obtain frequency text data by calculating the TF-IDF value of each keyword in the segmented text data, generate a keyword feature set based on a hash function and generate an identifier, thereby solving the problem of low efficiency in current text data deduplication.

[0032] As a preferred solution of the dynamic document deduplication system for big data environment described in the present invention, it includes: a keyword segmentation module, an identifier generation module, and a document deduplication module; the keyword segmentation module is used to collect original text data and preprocess it, and perform keyword segmentation on the preprocessed text data based on a natural language processing method; the identifier generation module is used to calculate the TF-IDF value of each keyword in the segmented text data to obtain frequency text data, generate a keyword feature set based on a hash function and generate an identifier; the document deduplication module is used to compare the similarity of each document through the identifier, and build a distributed computing framework to perform parallel deduplication processing on the comparison process.

[0033] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a dynamic document deduplication method for a big data environment.

[0034] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a dynamic document deduplication method for a big data environment.

[0035] Beneficial effects of the present invention: The dynamic document deduplication method for a big data environment provided by the present invention collects original text data and performs preprocessing, cleans and removes special characters, punctuation marks and stop words, and converts the text into lowercase letters, which can improve the accuracy and efficiency of subsequent processing. The preprocessed text data is segmented by keywords based on a natural language processing method, which further improves the structured level of the data and facilitates subsequent analysis and processing. The text data is converted into a compact feature vector through TF-IDF value calculation and hash function mapping, which reduces the dimension of the data while retaining important keyword information, thereby improving the efficiency and accuracy of data processing. Through parallel computing and task distribution, the advantages of distributed computing are fully utilized to achieve efficient deduplication processing in a large-scale data environment, and the efficiency bottleneck problem existing in the prior art when processing large-scale data is solved. The present invention achieves better results in terms of accuracy, efficiency and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0037] Figure 1 An overall flow chart of a dynamic document deduplication method for a big data environment provided for the first embodiment of the present invention.

[0038] Figure 2 An overall flow chart of a dynamic document deduplication system for a big data environment provided for the third embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0040] Example 1

[0041] Reference Figure 1 , is an embodiment of the present invention, and provides a dynamic document deduplication method for a big data environment, comprising:

[0042] S1: Collect and preprocess the original text data, and perform keyword segmentation on the preprocessed text data based on natural language processing methods.

[0043] Furthermore, preprocessing includes collecting raw text data, cleaning and converting the raw text data; the sources of raw text data include articles, news, blogs, and social media; cleaning includes removing special characters, punctuation marks, numbers, and stop words; conversion includes converting the raw text data into lowercase letters.

[0044] It should be noted that keyword segmentation includes performing word segmentation on preprocessed text data based on natural language processing methods, obtaining a word vector sequence, inputting the word vector sequence into a context encoder to obtain a word-granularity context semantic feature vector, inputting the word vector sequence into a bidirectional long short-term memory neural network model to obtain a segment-granularity context semantic feature vector, and fusing the word-granularity context semantic feature vector and the segment-granularity context semantic feature vector to obtain a multi-granularity fusion feature vector, which is expressed as:

[0045] V a =αV b +βV c

[0046] Among them, V a represents the multi-granularity fusion feature vector, α represents the weighting parameter of the word-granularity contextual semantic feature vector, β represents the weighting parameter of the segment-granularity contextual semantic feature vector, V b Represents the word-level contextual semantic feature vector, V cRepresenting the segment-granularity context semantic feature vector; constructing a corpus, generating a corpus feature vector of the preprocessed text data, obtaining the erroneous words and the position subscripts of the erroneous words based on the feature differences between the multi-granularity fusion feature vector and the corpus feature vector, and correcting the preprocessed text data through the position subscripts of the erroneous words.

[0047] It should also be noted that keyword segmentation and feature vector generation combine the semantic features of word granularity and segment granularity, and use multi-granularity fusion feature vectors to make text feature representation more comprehensive and accurate. Through the application of context encoders and Bi-LSTM models, the deep understanding of text semantics is improved, overcoming the limitations of single feature representation in traditional methods, building a corpus and generating corpus feature vectors for preprocessed text data, identifying and correcting erroneous words through feature differences with multi-granularity fusion feature vectors, and effectively improving the accuracy and consistency of text data through precise semantic feature matching. The application of multi-granularity fusion feature vectors makes error identification more accurate, thereby ensuring the efficiency and reliability of correction.

[0048] S2: Calculate the TF-IDF value of each keyword in the segmented text data to obtain frequency text data, generate a keyword feature set based on the hash function, and generate an identifier.

[0049] Furthermore, obtaining frequency text data includes obtaining segmented text data by keyword segmentation, and calculating the TF-IDF value of each keyword in the segmented text data, which is expressed as:

[0050] TF-IDF(t,d,D)=TF(t,d)×IDF(t,D)

[0051] Where t represents a keyword, d represents segmented text data, D represents a data set, TF-IDF(t,d,D) represents the TF-IDF value of keyword t in segmented text data d, TF(t,d) represents the frequency of keyword t in segmented text data d, IDF(t,D) represents the inverse document frequency of keyword t in data set D, and the calculation of inverse document frequency is expressed as:

[0052]

[0053] Among them, |D| represents the total number of segmented text data d in the dataset, and |{d∈D:t∈d| represents the number of segmented text data d containing keyword t.

[0054] It should be noted that generating the identifier includes storing the keywords and the corresponding frequencies in a hash table according to the frequency of occurrence of each keyword in the frequency text data, and sorting them from high to low according to the frequency of occurrence, setting a frequency threshold θ, and selecting keywords whose frequency of occurrence is greater than or equal to the frequency threshold, which is expressed as:

[0055] S={t∈T|TF(t,d)≥θ}

[0056] Among them, S represents the selected keyword set, and the feature information of the selected keywords is generated by the hash function to obtain the keyword feature set, which is expressed as:

[0057]

[0058] Among them, F is the keyword feature matrix, TF-IDF(t m ,d n ,D) indicates the keyword t m In segmenting text data n TF-IDF value in, m represents the number of keywords in the feature matrix, n represents the number of documents in the feature matrix, and the key features are mapped by hash functions. Each hash function maps the feature matrix to a hash value, which is expressed as:

[0059]

[0060] Among them, h k (F) is represented by the output of the kth hash function, a' ij represents the random constant that weights the elements in the i-th row and j-th column of the feature matrix, b' k represents the offset constant of the kth hash function, mod represents the modulus operation, p k Represents the prime number used by the kth hash function, normalizes the result of the hash function, and intercepts the binary bits of a fixed length to generate the final identifier, which is expressed as:

[0061]

[0062] Here, H(F) represents the final identifier, K represents the number of hash functions used, and |.| L It means taking binary bits of length L.

[0063] It should also be noted that by obtaining segmented text data through keyword segmentation and calculating the TF-IDF value of each keyword, the importance of each keyword in the document can be effectively evaluated, thereby laying the foundation for subsequent feature extraction and similarity calculation. The TF-IDF calculation combines word frequency and inverse document frequency, so that the weight of words that appear frequently but have low discrimination in multiple documents is reduced, and the keywords with discrimination are highlighted. In this way, the problem that the traditional method is easily affected by common words is solved, and the accuracy and effectiveness of keyword extraction are improved. According to the frequency of each keyword in the frequency text data, the keywords and the corresponding frequencies are stored in a hash table, and sorted from high to low according to the frequency of occurrence. A frequency threshold is set, and keywords with a frequency greater than or equal to the threshold are selected to generate a keyword feature set. This process effectively reduces the influence of unimportant or noise keywords by introducing a frequency threshold, making the generated feature set more refined and representative. The hash function maps the feature matrix to the hash value, and further generates a fixed-length binary identifier through normalization. This identifier can efficiently represent document features and facilitate subsequent similarity calculations.

[0064] S3: Compare the similarities of each document through identifiers and build a distributed computing framework to perform parallel deduplication processing on the comparison process.

[0065] Furthermore, comparing the similarity of each document includes assigning the identifiers to different computing nodes for parallel processing, and each computing node is responsible for calculating the similarity between the assigned identifier pairs, expressed as:

[0066]

[0067] Among them, X i'j' Represents the similarity between the i'th document and the j'th document, H(F) i' represents the final identifier of the i'th document, H(F) j' represents the final identifier of the j'th document, H(F) i'k Indicates the kth bit of the final identifier of the i'th document, H(F) j'k Represents the kth bit of the final identifier of the j'th document.

[0068] It should be noted that the parallel deduplication processing includes building a distributed computing framework to perform parallel deduplication processing on the comparison process. Each computing node performs deduplication judgment on the responsible similarity matrix according to a preset similarity threshold, which is expressed as:

[0069]

[0070] Among them, R i'j'represents the deduplication result between the i'th document and the j'th document, τ represents the similarity threshold, when the similarity of two documents is X i'j' When the similarity X between two documents is greater than or equal to the threshold τ, the two documents are duplicates and need to be deduplicated. i'j' When it is less than the threshold τ, the two documents are not duplicates and do not need to be deduplicated, so the documents are retained.

[0071] It should also be noted that the identifiers are assigned to different computing nodes for parallel processing. Each computing node is responsible for calculating the similarity between the assigned identifier pairs. By building a distributed computing framework, the parallelization of similarity calculation is achieved, which improves the computing efficiency. By setting a similarity threshold, documents with similarity higher than the threshold are marked as duplicate documents and deduplication is performed. Through the parallel processing of the distributed computing framework, the deduplication task of large-scale document data can be completed quickly and efficiently, solving the problems of slow processing speed and low accuracy in the prior art.

[0072] Example 2

[0073] An embodiment of the present invention provides a dynamic document deduplication method for a big data environment. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0074] First, in order to verify the effectiveness of the dynamic document deduplication method of the present invention in a big data environment, original text data from multiple sources, including articles, news, blogs and social media data, are selected. First, these original text data are preprocessed, including cleaning and conversion. The cleaning step includes removing special characters, punctuation marks, numbers, and stop words. The conversion step converts all text data into lowercase letters to ensure the consistency and comparability of the data. Next, the preprocessed text data is segmented by keywords based on the natural language processing method. Specifically, word segmentation is used to obtain a word vector sequence, and the word vector sequence is input into a context encoder to obtain a word-granularity context semantic feature vector. At the same time, the word vector sequence is input into a bidirectional long short-term memory neural network model (Bi-LSTM) to obtain a segment-granularity context semantic feature vector. Finally, the word-granularity and segment-granularity context semantic feature vectors are fused to obtain a multi-granularity fused feature vector. After the multi-granularity fused feature vector is generated, a corpus is constructed and a preprocessed text data is generated. The corpus feature vector of the data is obtained, and the feature difference is calculated to obtain the wrong words and their position subscripts, and the preprocessed text data is corrected. Then, the segmented text data is obtained by keyword segmentation, and the TF-IDF value of each keyword is calculated to obtain the frequency text data. According to the frequency of occurrence of each keyword in the frequency text data, the keywords and the corresponding frequencies are stored in the hash table, and sorted from high to low by frequency. The frequency threshold is set to 0.05, and a set of keywords with an occurrence frequency greater than or equal to the threshold is selected. The keyword features are mapped through the hash function to generate a keyword feature set and an identifier. The similarity of each document is compared through the identifier, and a distributed computing framework is constructed to perform parallel deduplication processing on the comparison process. Each computing node is responsible for calculating the similarity between the assigned identifier pairs. According to the preset similarity threshold, the similarity matrix is ​​deduplicated. When the similarity of two documents is greater than or equal to the threshold, the two documents are considered to be duplicates and need to be deduplicated; otherwise, the documents are retained. Refer to Table 1 to record and analyze the experimental data.

[0075] Table 1 Experimental data record table

[0076]

[0077]

[0078] It can be seen from the data in the table that in the text preprocessing process, the original text data is cleaned and converted to effectively remove irrelevant information and noise data, making the subsequent keyword extraction and TF-IDF calculation more accurate. The table data shows that after the original text data is cleaned, the text length is reduced. For example, document 1 is reduced from 1500 words to 1200 words, and document 2 is reduced from 1800 words to 1450 words, which shows the effectiveness of data cleaning. Secondly, keyword segmentation and TF-IDF value calculation based on natural language processing methods can more accurately evaluate the importance of each keyword in the document. The number of keywords and average TF-IDF value in the table show the importance of keywords in different documents. For example, document 3 has the highest number of keywords (300) and the highest average TF-IDF value (0.007), indicating that there are more important keywords in the document, and parallel deduplication is performed in the distributed computing framework. The redundancy of documents is determined by calculating similarity scores. The similarity scores in the table show the similarity between different documents. For example, the similarity scores of Document 1 and Document 2 are 0.95 and 0.92 respectively, both of which are higher than the preset similarity threshold of 0.8, indicating that they are duplicate documents and need to be deduplicated. The similarity score of Document 3 is 0.15, which is much lower than the threshold of 0.8, indicating that its content is unique and needs to be retained. It can be seen from the above data analysis that the dynamic document deduplication method of the present invention in a big data environment can efficiently and accurately remove redundant documents, thereby improving the efficiency and accuracy of data processing. Compared with the prior art, the present invention is not only innovative in text preprocessing, keyword extraction and TF-IDF calculation, but also shows obvious technical advantages in hash function identifier generation and parallel deduplication processing. These innovations and improvements enable the present invention to better adapt to large-scale data environments and achieve efficient and accurate dynamic document deduplication.

[0079] Example 3

[0080] Reference Figure 2 , which is an embodiment of the present invention, provides a dynamic document deduplication system for a big data environment, including a keyword segmentation module, an identifier generation module, and a document deduplication module.

[0081] The keyword segmentation module is used to collect and preprocess the original text data, and perform keyword segmentation on the preprocessed text data based on the natural language processing method; the identifier generation module is used to calculate the TF-IDF value of each keyword in the segmented text data to obtain frequency text data, generate a keyword feature set based on the hash function and generate an identifier; the document deduplication module is used to compare the similarity of each document through the identifier, and build a distributed computing framework to perform parallel deduplication processing on the comparison process.

[0082] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0083] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0084] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0085] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0086] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A dynamic document deduplication method for a big data environment, characterized in that: include: Collect and preprocess the original text data, and perform keyword segmentation on the preprocessed text data based on natural language processing methods; Calculate the TF-IDF value of each keyword in the segmented text data to obtain frequency text data, generate a keyword feature set based on the hash function, and generate an identifier; The similarities of each document are compared through identifiers, and a distributed computing framework is built to perform parallel deduplication processing on the comparison process.

2. The method for deduplicating dynamic documents in a big data environment as claimed in claim 1, characterized in that: The preprocessing includes collecting original text data, cleaning and converting the original text data; The original text data sources include articles, news, blogs, and social media; Cleaning includes removing special characters, punctuation marks, numbers, and stop words; The transformation consists of converting the original text data to lowercase letters.

3. The method for deduplicating dynamic documents in a big data environment as claimed in claim 2, characterized in that: The keyword segmentation includes performing word segmentation processing on the preprocessed text data based on a natural language processing method, obtaining a word vector sequence, inputting the word vector sequence into a context encoder to obtain a word-granularity context semantic feature vector, inputting the word vector sequence into a bidirectional long short-term memory neural network model to obtain a segment-granularity context semantic feature vector, fusing the word-granularity context semantic feature vector and the segment-granularity context semantic feature vector to obtain a multi-granularity fusion feature vector, which is expressed as: V a =αV b +βV c Among them, V a represents the multi-granularity fusion feature vector, α represents the weighting parameter of the word-granularity contextual semantic feature vector, β represents the weighting parameter of the segment-granularity contextual semantic feature vector, V b Represents the word-level contextual semantic feature vector, V c Represents the segment-granular contextual semantic feature vector; A corpus is constructed to generate a corpus feature vector of the preprocessed text data. Based on the feature difference between the multi-granularity fusion feature vector and the corpus feature vector, the erroneous words and the position subscripts of the erroneous words are obtained, and the preprocessed text data is corrected through the position subscripts of the erroneous words.

4. The method for deduplicating dynamic documents in a big data environment as claimed in claim 3, characterized in that: The obtaining of frequency text data includes obtaining segmented text data by keyword segmentation, and calculating the TF-IDF value of each keyword in the segmented text data, which is expressed as: TF-IDF(t,d,D)=TF(t,d)×IDF(t,D) Where t represents a keyword, d represents segmented text data, D represents a data set, TF-IDF(t,d,D) represents the TF-IDF value of keyword t in segmented text data d, TF(t,d) represents the frequency of keyword t in segmented text data d, IDF(t,D) represents the inverse document frequency of keyword t in data set D, and the calculation of inverse document frequency is expressed as: Among them, |D| represents the total number of segmented text data d in the dataset, and |{d∈D:t∈d| represents the number of segmented text data d containing keyword t.

5. The method for deduplication of dynamic documents in a big data environment as claimed in claim 4, characterized in that: The generating identifier includes storing the keywords and the corresponding frequencies in a hash table according to the frequency of occurrence of each keyword in the frequency text data, and sorting them from high to low according to the frequency of occurrence, setting a frequency threshold θ, and selecting keywords whose frequency of occurrence is greater than or equal to the frequency threshold, which is expressed as: S={t∈T|TF(t,d)≥θ Among them, S represents the selected keyword set, and the feature information of the selected keywords is generated by the hash function to obtain the keyword feature set, which is expressed as: Among them, F is the keyword feature matrix, TF-IDF(t m ,d n ,D) indicates the keyword t m In segmenting text data n TF-IDF value in, m represents the number of keywords in the feature matrix, n represents the number of documents in the feature matrix, and the key features are mapped by hash functions. Each hash function maps the feature matrix to a hash value, which is expressed as: Among them, h k (F) is represented by the output of the kth hash function, a' ij represents the random constant that weights the elements in the i-th row and j-th column of the feature matrix, b' k represents the offset constant of the kth hash function, mod represents the modulus operation, p k Represents the prime number used by the kth hash function, normalizes the result of the hash function, and intercepts the binary bits of a fixed length to generate the final identifier, which is expressed as: Here, H(F) represents the final identifier, K represents the number of hash functions used, and |.| L It means taking binary bits of length L.

6. The method for deduplicating dynamic documents in a big data environment as claimed in claim 5, characterized in that: The comparison of the similarity of each document includes assigning the identifiers to different computing nodes for parallel processing, and each computing node is responsible for calculating the similarity between the assigned identifier pairs, which is expressed as: Among them, X i'j' Represents the similarity between the i'th document and the j'th document, H(F) i' represents the final identifier of the i'th document, H(F) j' represents the final identifier of the j'th document, H(F) i'k Indicates the kth bit of the final identifier of the i'th document, H(F) j'k Represents the kth bit of the final identifier of the j'th document.

7. The method for deduplicating dynamic documents in a big data environment as claimed in claim 6, characterized in that: The parallel deduplication processing includes constructing a distributed computing framework to perform parallel deduplication processing on the comparison process. Each computing node performs deduplication judgment on the responsible similarity matrix according to a preset similarity threshold, which is expressed as: Among them, R i'j' represents the deduplication result between the i'th document and the j'th document, τ represents the similarity threshold, when the similarity of two documents is X i'j' When it is greater than or equal to the threshold τ, the two documents are duplicates and need to be deduplicated; When the similarity X between two documents i'j' When it is less than the threshold τ, the two documents are not duplicates and do not need to be deduplicated, so the documents are retained.

8. A system using the dynamic document deduplication method for a big data environment as described in any one of claims 1 to 7, characterized in that: Including keyword segmentation module, identifier generation module, document deduplication module; The keyword segmentation module is used to collect and preprocess the original text data, and perform keyword segmentation on the preprocessed text data based on a natural language processing method; The identifier generation module is used to calculate the TF-IDF value of each keyword in the segmented text data to obtain frequency text data, generate a keyword feature set based on a hash function, and generate an identifier; The document deduplication module is used to compare the similarity of each document through identifiers and build a distributed computing framework to perform parallel deduplication processing on the comparison process.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the dynamic document deduplication method for a big data environment described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the dynamic document deduplication method for a big data environment described in any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Document segmentation method for large model retrieval enhancement generation

    CN119848188A