Intelligent duplicate checking detection system and method based on multi-feature mimicry of target entity

Abstract

The invention relates to the technical field of data detection, and discloses an intelligent duplicate checking detection system and method based on target entity multi-feature mimicry, and the method comprises the steps: collecting to-be-detected target entity data, and carrying out the word segmentation of the to-be-detected target entity data based on natural language processing; extracting text features and semantic features; the features are fused to obtain mimicry features, the mimicry features are compared with a historical database, a duplicate checking file is determined, and duplicate checking similarity is determined; comparing the duplicate checking similarity with a similarity threshold to judge whether the duplicate checking similarity is adjusted or not; and when the similarity is smaller than a similarity threshold, judging to carry out adjustment, processing the data after word segmentation based on a graph neural network to extract structural features, comparing the structural features with historical structural features in a historical database, and determining an adjustment coefficient according to a comparison result to adjust the duplicate checking similarity. According to the method, the duplicate checking accuracy is improved, the similarity of the target text is comprehensively evaluated, and the duplicate checking precision and robustness are improved.

Description

Technical Field

[0001] The present invention relates to the field of data detection technology, and in particular to an intelligent duplicate detection system and method based on multi-feature mimicry of a target entity. Background Art

[0002] With the rapid development of information technology, the amount of text data generated and disseminated has increased year by year. Especially in the fields of academic research, document management and intellectual property, the importance of duplicate detection technology has become increasingly prominent. Current duplicate detection is usually based on direct comparison of texts, using string matching algorithms or text similarity calculation methods based on word frequency statistics. These methods can effectively identify duplicate content in some scenarios, but with the development of semantic analysis technology, traditional duplicate detection methods have gradually exposed some shortcomings.

[0003] At present, duplicate checking methods based on natural language processing (NLP) technology are gradually gaining attention, and the accuracy of duplicate checking is improved by introducing semantic understanding and context analysis. However, existing methods still have certain limitations, for example: the extraction of semantic features often relies on pre-trained models, which cannot fully combine the specific content of the target text; existing duplicate checking methods mostly rely on single features (such as text or semantic features) for comparison, lacking a comprehensive analysis of the multi-dimensional and multi-level features of the target entity; lacking the mining of text structure features, and unable to make dynamic adjustments based on differences in text structure, resulting in inaccurate calculation of duplicate checking similarity.

[0004] Therefore, it is necessary to design an intelligent duplicate detection system and method based on multi-feature mimicry of the target entity to solve the problems existing in the current technology. Summary of the invention

[0005] In view of this, the present invention proposes an intelligent duplicate detection system and method based on multi-feature mimicry of target entities, aiming to solve the problem that current duplicate detection methods cannot fully combine the specific content of the target text and lack multi-dimensional analysis of the target entity, resulting in inaccurate similarity calculation.

[0006] In one aspect, the present invention proposes an intelligent duplicate detection method based on multi-feature mimicry of a target entity, comprising: Collecting target entity data to be detected, and performing word segmentation on the target entity data to be detected based on natural language processing; The data after word segmentation is processed based on the bag-of-words model to extract text features, and the data after word segmentation is processed based on BERT (Bidirectional Encoder Representations from Transformers pre-trained language model) to extract semantic features; The text features and semantic features are integrated to obtain mimetic features, and the mimetic features are compared with the historical database to determine the duplicate check files and determine the duplicate check similarity; The duplicate check similarity is compared with a similarity threshold, and whether to adjust the duplicate check similarity is determined according to the comparison result; when the duplicate check similarity is less than the similarity threshold, it is determined to adjust the duplicate check similarity, and the data after word segmentation is processed based on a graph neural network to extract structural features, and the structural features are compared with the historical structural features in the historical database, and an adjustment coefficient is determined according to the comparison result to adjust the duplicate check similarity.

[0007] Furthermore, when the target entity data to be detected is segmented based on natural language processing, it includes: Performing data cleaning on the target entity data to be detected, wherein the data cleaning includes removing stop words and punctuation marks and converting English text into lowercase letters; Perform word count on the cleaned data, including the number of Chinese characters and the number of English characters, and determine the word segmentation tool according to the ratio of the number of Chinese characters to the number of English characters; When the ratio of the number of Chinese characters to the number of English characters is greater than 1, the word segmentation tools include Jieba, THULAC and HanLP; When the ratio of the number of Chinese characters to the number of English characters is less than or equal to 1, the word segmentation tools include NLTK, spaCy and StanfordNLP.

[0008] Furthermore, when the data after word segmentation is processed based on the bag-of-words model to extract text features, it includes: Pre-build a vocabulary V={w1,w2,...,wn}, where Wn represents the nth word in the vocabulary; Calculate the number of occurrences of each word in the word segmented data in the vocabulary; All occurrence times of the segmented data are represented by feature vectors to obtain the text features.

[0009] Furthermore, when the text features and semantic features are integrated to obtain mimicry features, it includes: The text features and the semantic features are adjusted to the same dimension by using a zero-filling method; The text features and semantic features are fused by weighted fusion to obtain mimicry features; ; Among them, V represents the mimicry feature, Vw represents the text feature, Vy represents the semantic feature, a1 and a2 represent the weight coefficients, and a1+a2=1.

[0010] Furthermore, when comparing the mimicry feature with the historical database to determine the duplicate check file and determine the duplicate check similarity, it includes: The historical database includes historical mimic features of several historical detection target entity data; Calculating the similarity between the mimicry feature and each historical mimicry feature; ; Among them, Si represents the similarity between the mimicry feature and the i-th historical mimicry feature, V represents the mimicry feature, Vsi represents the i-th historical mimicry feature, Represents the magnitude of a vector.

[0011] Further, judging whether to adjust the similarity of the duplicate check according to the comparison result includes: When the similarity of the duplicate check is greater than or equal to the similarity threshold, it is determined that the similarity of the duplicate check is not to be adjusted, and the duplicate check file and the similarity of the duplicate check are output; When the duplicate checking similarity is less than the similarity threshold, it is determined to adjust the duplicate checking similarity.

[0012] Further, when determining to adjust the similarity of the duplicate check, and processing the segmented data based on the graph neural network to extract structural features, it includes: Convert the segmented data into a graph structure, wherein nodes in the graph structure represent basic elements, including words or sentences, and edges in the graph structure represent relationships between elements; Use Word2Vec to map each of the basic elements to a vector of fixed dimension; Update the representation of nodes in each layer based on convolution operations; ; in, represents the representation of node v at the k+1th layer; Represents the set of neighbor nodes of node v; represents the weight matrix of the kth layer; represents the bias term, represents the activation function, Indicates the representation of node u at layer k; The structural features are obtained by performing average pooling on the representations of all nodes.

[0013] Furthermore, when the adjustment coefficient is determined according to the comparison result to adjust the similarity of the duplicate check, it includes: Selecting the historical mimicry feature corresponding to the maximum value of the similarity between the mimicry feature and the historical mimicry feature, and obtaining the historical structural feature of the historical mimicry feature, obtaining the modulus difference between the structural feature and the historical structural feature, and determining an adjustment coefficient according to the modulus difference to adjust the similarity; The modulus difference is compared with a first preset modulus difference and a second preset modulus difference respectively, and an adjustment coefficient is determined according to the comparison result to adjust the duplicate similarity; the first preset modulus difference is smaller than the second preset modulus difference; When the modulus difference is less than or equal to a first preset modulus difference, a first adjustment coefficient is determined to adjust the duplicate similarity; when the modulus difference is greater than the first preset modulus difference and less than or equal to a second preset modulus difference, a second adjustment coefficient is determined to adjust the duplicate similarity; when the modulus difference is greater than the second preset modulus difference, a third adjustment coefficient is determined to adjust the duplicate similarity; the first adjustment coefficient is greater than the second adjustment coefficient, the second adjustment coefficient is greater than the third adjustment coefficient, and the adjustment coefficient takes a value range of (0, 1), and the adjusted duplicate similarity is the product of the duplicate similarity and the adjustment coefficient.

[0014] Compared with the prior art, the beneficial effect of the present invention is that by introducing a multi-feature fusion method, combined with natural language processing, deep learning and graph neural network technologies, the intelligence and accuracy of the duplicate checking system are improved. Based on the word segmentation and word bag model of natural language processing, text features are extracted, and then the semantic features are extracted in combination with the BERT model, and the text of the target entity is analyzed from multiple levels and dimensions, avoiding the limitations of the single feature comparison of the traditional method, and being able to more comprehensively understand and capture the potential similarity of the text. The text features are fused with the semantic features to generate mimetic features, and then the similarity of the duplicate checking is accurately calculated by comparing with the historical database, thereby achieving more efficient duplicate checking detection. When the duplicate checking similarity is lower than the set threshold, the graph neural network is further introduced to mine and compare the text structure features, and the structural differences of the text are dealt with by adjusting the similarity, avoiding the duplicate checking errors caused by structural differences. The accuracy of the duplicate checking is improved, the similarity of the target text is comprehensively evaluated, and the accuracy and robustness of the duplicate checking are improved.

[0015] On the other hand, the present application also provides an intelligent duplicate detection system based on multi-feature mimicry of a target entity, which is used to apply the above-mentioned intelligent duplicate detection method based on multi-feature mimicry of a target entity, including: A collection unit is configured to collect target entity data to be detected, and perform word segmentation on the target entity data to be detected based on natural language processing; A processing unit is configured to process the segmented data based on a bag-of-words model to extract text features, and to process the segmented data based on BERT to extract semantic features; A judgment unit is configured to fuse the text features and the semantic features to obtain a mimetic feature, compare the mimetic feature with a historical database, determine a duplicate check file, and determine a duplicate check similarity; The adjustment unit is configured to compare the duplicate check similarity with a similarity threshold, and determine whether to adjust the duplicate check similarity according to the comparison result; when the duplicate check similarity is less than the similarity threshold, determine to adjust the duplicate check similarity, and process the segmented data based on a graph neural network to extract structural features, compare the structural features with the historical structural features in the historical database, and determine an adjustment coefficient according to the comparison result to adjust the duplicate check similarity.

[0016] Furthermore, when the adjustment unit determines the adjustment coefficient according to the comparison result to adjust the similarity of the duplicate check, it also includes: The adjustment unit selects the historical mimicry feature corresponding to the maximum value of the duplicate similarity between the mimicry feature and the historical mimicry feature, obtains the historical structural feature of the historical mimicry feature, obtains the modulus difference between the structural feature and the historical structural feature, and determines the adjustment coefficient according to the modulus difference to adjust the duplicate similarity.

[0017] It can be understood that the above-mentioned intelligent duplicate detection system and method based on multi-feature mimicry of the target entity have the same beneficial effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 A flowchart of an intelligent duplicate detection method based on multi-feature mimicry of a target entity provided by an embodiment of the present invention; Figure 2 A functional block diagram of an intelligent duplicate detection system based on multi-feature mimicry of a target entity provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0020] In some embodiments of the present application, see Figure 1 As shown, an intelligent duplicate detection method based on multi-feature mimicry of a target entity includes: S100: Collect target entity data to be detected, and perform word segmentation on the target entity data to be detected based on natural language processing.

[0021] S200: Process the segmented data based on the bag-of-words model to extract text features, and process the segmented data based on BERT to extract semantic features.

[0022] S300: The text features and semantic features are integrated to obtain mimetic features, and the mimetic features are compared with the historical database to determine the duplicate files and determine the duplicate similarity.

[0023] S400: Compare the duplicate similarity with the similarity threshold, and determine whether to adjust the duplicate similarity based on the comparison result. When the duplicate similarity is less than the similarity threshold, it is determined to adjust the duplicate similarity, and the data after word segmentation is processed based on the graph neural network to extract structural features, and the structural features are compared with the historical structural features in the historical database. According to the comparison result, the adjustment coefficient is determined to adjust the duplicate similarity.

[0024] Specifically, the target text data to be detected is collected in S100. Through natural language processing technology, the text data is segmented and divided into basic language units (such as words, phrases, etc.), providing a basis for subsequent feature extraction and analysis. Convert text information into structured data that can be understood by the machine. In S200, the bag-of-words model is used to extract text features on the data after word segmentation. The bag-of-words model helps capture basic information in the text, such as the repetition and distribution pattern of vocabulary by counting the frequency of occurrence of each word in the text. The semantic features of the text are extracted using the BERT model. BERT is a deep learning language model that can better understand the context and thus capture the deep semantic information in the text. This embodiment can not only identify the literal similarity of the text, but also understand the meaning and context of the text, thereby improving the accuracy of the duplicate check. In S300, the text features extracted by the bag-of-words model and the semantic features extracted by BERT are fused to generate a "mimetic feature". The mimetic feature combines the surface information and deep semantic information of the text, representing the multi-dimensional features of the target entity. Compare the mimetic features with the content in the historical database to identify existing similar texts. By calculating the similarity of the duplicate check, preliminarily determine the similarity between the text and the historical data. In S400, based on the preliminary duplicate check, the duplicate check similarity is compared with the set similarity threshold. If the duplicate check similarity is lower than the threshold, further adjustments are made. The structural features of the text are extracted based on the graph neural network (GNN Graph Neural Network) to analyze the structure and organization of the text. The potential structural similarities of the text (such as paragraphs, sentence structures, etc.) are captured through the graph neural network and compared with the historical structural features in the historical database.

[0025] By comparing the results, the similarity of the duplicate check is adjusted according to the similarity of the text structure, so as to more accurately evaluate the degree of repetition of the text. The adjusted similarity can more effectively identify the true similarity of the text and avoid misjudgment and omission.

[0026] It is understandable that by integrating text features, semantic features and structural features, and adopting advanced technologies such as natural language processing, BERT model and graph neural network, the accuracy and robustness of duplicate detection have been improved. It can simultaneously consider the surface features, semantic levels and differences in text structure of the text, avoiding the limitations of traditional methods that rely on a single feature, and can more comprehensively identify the similarity of complex texts; semantic feature extraction based on deep learning and structural feature analysis of graph neural networks enable the system to dynamically adjust the similarity of duplicate detection, reducing misjudgments and missed judgments caused by factors such as text rewriting and synonym replacement; multi-dimensional feature fusion and dynamic adjustment mechanism enable duplicate detection to respond to different types of text more intelligently, with higher accuracy and flexibility than traditional duplicate detection methods.

[0027] In some embodiments of the present application, when performing word segmentation on the target entity data to be detected based on natural language processing, it includes: cleaning the data of the target entity data to be detected, and the data cleaning includes removing stop words and punctuation marks, and converting English text into lowercase letters.

[0028] Count the number of words in the cleaned data, including the number of Chinese characters and the number of English words, and determine the word segmentation tool according to the ratio of the number of Chinese characters to the number of English words.

[0029] When the ratio of the number of Chinese characters to the number of English words is greater than 1, the word segmentation tools include Jieba, THULAC, and HanLP.

[0030] When the ratio of the number of Chinese characters to the number of English words is less than or equal to 1, the word segmentation tools include NLTK, spaCy, and StanfordNLP.

[0031] It can be understood that stop words refer to words that frequently appear in the text but contribute little to the meaning of the text (such as "de", "he", "shi", etc.). Removing these stop words helps to reduce noise and improve the effectiveness of subsequent analysis. Punctuation marks are important for the transmission of word meanings, but they are usually not used as independent units when performing word segmentation, so they need to be removed. By counting the number of words in the target entity data and analyzing the language ratio, the most suitable word segmentation tool is selected, thereby ensuring the efficiency and accuracy of word segmentation in different language environments. By flexibly adjusting the word segmentation tool used according to the language characteristics of the text, the limitations in the application of traditional single word segmentation tools are effectively avoided, and complex texts containing mixed languages (such as Chinese-English mixed) can be handled. The adaptability and accuracy of text preprocessing are improved. For texts containing a large amount of English or Chinese-English mixed, they can be processed more accurately, thereby improving the overall effect of subsequent text feature extraction, semantic analysis, and duplicate checking calculation. The applicability and accuracy of the duplicate checking detection system in multi-language and multi-text type environments are enhanced.

[0032] In some embodiments of the present application, when processing the segmented data based on the bag-of-words model to extract text features, it includes: pre-constructing a vocabulary V = {w1, w2,..., wn}, where Wn represents the nth word in the vocabulary. Calculate the number of occurrences of each word in the segmented data in the vocabulary. Represent the number of occurrences of all the segmented data as a feature vector to obtain text features.

[0033] In some embodiments of the present application, when fusing the text features and semantic features to obtain mimetic features, it includes: Using the zero-padding method to adjust the text features and semantic features to the same dimension.

[0034] ; Among them, V represents the mimicry feature, Vw represents the text feature, Vy represents the semantic feature, a1 and a2 represent the weight coefficients, and a1+a2=1.

[0035] It is understandable that the capability of the duplicate detection system is improved through feature extraction and fusion. By extracting features from the text through the bag-of-words model, the distribution of words in the text can be fully captured, providing basic information for subsequent duplicate checking. The zero-filling method is used to ensure the consistency of text features and semantic features in dimensions, so that the two can be smoothly integrated. Through weighted fusion, the relative importance of text features and semantic features can be flexibly adjusted according to specific needs, so as to obtain the best duplicate checking effect in different application scenarios. It can optimize the accuracy of duplicate checking at multiple feature levels (text, semantics) at the same time, improve adaptability and accuracy, and maintain efficient processing capabilities when facing different types of text (for example, documents with rich semantic information or texts with complex language structures).

[0036] In some embodiments of the present application, when comparing the mimicry feature with the historical database, determining the duplicate check file and determining the duplicate check similarity, it includes: The historical database includes historical mimic features of several historical detection target entity data.

[0037] Calculate the similarity between the mimicry feature and each historical mimicry feature.

[0038] ; Among them, Si represents the similarity between the mimicry feature and the i-th historical mimicry feature, V represents the mimicry feature, Vsi represents the i-th historical mimicry feature, Represents the magnitude of a vector.

[0039] In some embodiments of the present application, when determining whether to adjust the similarity of the duplicate check according to the comparison result, it includes: when the similarity of the duplicate check is greater than or equal to the similarity threshold, it is determined that the similarity of the duplicate check is not adjusted, and the duplicate check file and the similarity of the duplicate check are output. When the similarity of the duplicate check is less than the similarity threshold, it is determined that the similarity of the duplicate check is adjusted.

[0040] It is understandable that, assuming an article to be detected, and the mimicry features of several articles that have been detected are stored in the historical database. First, the mimicry features of the article to be detected are calculated, and the similarity is calculated with the mimicry features of each article in the historical database. The similarity between the mimicry features of the article to be detected and the mimicry features of the 5th article in the historical database is 0.85, and the similarity with the 10th article is 0.65. The similarity threshold is set to 0.8, and the similarity between the 5th article and the article to be detected is judged to be higher than the threshold. The duplicate check result is directly output and marked as high similarity; the similarity of the 10th article is lower than the threshold, and the similarity will be adjusted based on further analysis, and more text structure features will be considered to optimize or adjust the similarity.

[0041] By calculating the similarity between the mimetic features and the historical mimetic features in the historical database, texts with high similarity can be identified, avoiding the limitations of simple literal matching or surface feature-based duplicate detection methods. The introduction of similarity threshold judgment and dynamic adjustment mechanism improves the flexibility and accuracy of the duplicate detection system. When faced with texts with low similarity or potential subtle similarities, further adjustments are made to optimize the duplicate detection results to avoid missed or misjudgment. The multi-stage duplicate detection and similarity adjustment process can cope with complex texts and provide more accurate and comprehensive duplicate detection.

[0042] In some embodiments of the present application, when determining to adjust the similarity of duplicate checking, and extracting structural features from the segmented data based on a graph neural network, the process includes: The segmented data is converted into a graph structure, in which nodes represent basic elements, including words or sentences, and edges represent the relationship between elements.

[0043] Use Word2Vec to map each basic element to a vector of fixed dimension.

[0044] The representation of nodes is updated in each layer based on convolution operations.

[0045] ; in, represents the representation of node v at the k+1th layer; Represents the set of neighbor nodes of node v; represents the weight matrix of the kth layer; represents the bias term, represents the activation function, Represents the representation of node u at the kth layer.

[0046] The representations of all nodes are averaged and pooled to obtain structural features.

[0047] In some embodiments of the present application, when determining an adjustment coefficient based on the comparison result to adjust the duplicate similarity, it includes: selecting the historical mimicry feature corresponding to the maximum value of the duplicate similarity between the mimicry feature and the historical mimicry feature, and obtaining the historical structural feature of the historical mimicry feature, obtaining the modulus difference between the structural feature and the historical structural feature, and determining the adjustment coefficient based on the modulus difference to adjust the duplicate similarity. The modulus difference is compared with the first preset modulus difference and the second preset modulus difference respectively, and the adjustment coefficient is determined based on the comparison result to adjust the duplicate similarity. The first preset modulus difference is less than the second preset modulus difference.

[0048] Specifically, when the modulus difference is less than or equal to the first preset modulus difference, the first adjustment coefficient is determined to adjust the similarity of the duplicate check. When the modulus difference is greater than the first preset modulus difference and less than or equal to the second preset modulus difference, the second adjustment coefficient is determined to adjust the similarity of the duplicate check. When the modulus difference is greater than the second preset modulus difference, the third adjustment coefficient is determined to adjust the similarity of the duplicate check. The first adjustment coefficient is greater than the second adjustment coefficient, the second adjustment coefficient is greater than the third adjustment coefficient, and the adjustment coefficient value range is (0, 1), and the adjusted similarity of the duplicate check is the product of the similarity of the duplicate check and the adjustment coefficient.

[0049] Specifically, the data after word segmentation is converted into a graph structure. The nodes in the graph represent the basic elements in the text. The basic elements can be words, phrases, or sentences. The edges in the graph represent the relationship between these elements, which are usually defined based on grammatical, semantic, or contextual information. For example, an edge can represent the dependency between words or the semantic connection between sentences. The Word2Vec model is used to map each basic element (such as a word or sentence) into a vector space of fixed dimension. These vectors can retain the semantic similarity between elements, so the representation of the node can reflect the semantic content of the elements in the text. The convolution operation in the graph neural network (Graph Convolutional Network, GCN) is used to update the node representation of each layer. At each layer, the representation of the node is affected by its neighboring nodes. After updating the node representation through multi-layer convolution operations, the representation of all nodes is averaged and pooled to summarize the representation of each node into a global structural feature, reflecting the overall structural information of the text. These structural features provide additional contextual information in the duplicate checking process, which helps to distinguish texts that are similar on the surface but different in structure.

[0050] It is understandable that by converting text into a graph structure and using graph neural networks to capture the grammatical, semantic, and structural information in the text, it is possible to more accurately identify the deep similarities and differences between texts when checking for duplicate content. Through the mechanism of modulus difference and adjustment coefficient, the similarity of duplicate content checking is dynamically adjusted according to the structural differences of the text, making the final duplicate content checking results more accurate. Similarity adjustment not only improves the accuracy of duplicate content checking, but also avoids over-reliance on surface features, and enhances robustness when facing texts with complex structures and diverse grammars.

[0051] In the above embodiments, by introducing a method of multi-feature fusion, combined with natural language processing, deep learning and graph neural network technologies, the intelligence and accuracy of the duplicate checking system are improved. Based on the word segmentation and word bag model of natural language processing, text features are extracted, and then the semantic features are extracted in combination with the BERT model, and the text of the target entity is analyzed from multiple levels and dimensions, avoiding the limitations of the single feature comparison of the traditional method, and being able to more comprehensively understand and capture the potential similarity of the text. The text features are fused with the semantic features to generate mimetic features, and then the duplicate checking similarity is accurately calculated by comparing with the historical database, thereby achieving more efficient duplicate checking detection. When the duplicate checking similarity is lower than the set threshold, the graph neural network is further introduced to mine and compare the text structure features, and the structural differences of the text are dealt with by adjusting the similarity, avoiding the duplicate checking errors caused by structural differences. The accuracy of the duplicate checking is improved, the similarity of the target text is comprehensively evaluated, and the accuracy and robustness of the duplicate checking are improved.

[0052] In another preferred embodiment based on the above embodiment, refer to Figure 2 As shown, this embodiment provides an intelligent duplicate detection system based on multi-feature mimicry of a target entity, which is used to apply the above-mentioned intelligent duplicate detection method based on multi-feature mimicry of a target entity, including: The collection unit is configured to collect target entity data to be detected and perform word segmentation on the target entity data to be detected based on natural language processing.

[0053] The processing unit is configured to process the segmented data based on the bag-of-words model to extract text features, and to process the segmented data based on BERT to extract semantic features.

[0054] The judgment unit is configured to fuse the text features and the semantic features to obtain the mimetic features, compare the mimetic features with the historical database, determine the duplicate checking files and determine the duplicate checking similarity.

[0055] The adjustment unit is configured to compare the duplicate similarity with the similarity threshold, and determine whether to adjust the duplicate similarity according to the comparison result. When the duplicate similarity is less than the similarity threshold, it is determined to adjust the duplicate similarity, and the data after word segmentation is processed based on the graph neural network to extract structural features, and the structural features are compared with the historical structural features in the historical database. According to the comparison result, the adjustment coefficient is determined to adjust the duplicate similarity.

[0056] Furthermore, when the adjustment unit determines the adjustment coefficient according to the comparison result to adjust the similarity of the duplicate check, it also includes: The adjustment unit selects the historical mimicry feature corresponding to the maximum value of the similarity between the mimicry feature and the historical mimicry feature, obtains the historical structural feature of the historical mimicry feature, obtains the modulus difference between the structural feature and the historical structural feature, and determines the adjustment coefficient according to the modulus difference to adjust the similarity.

[0057] It is understandable that by introducing the method of multi-feature fusion, combined with natural language processing, deep learning and graph neural network technologies, the intelligence and accuracy of the duplicate checking system have been improved. Based on the word segmentation and bag-of-words model of natural language processing, text features are extracted, and then combined with the BERT model to extract semantic features, the text of the target entity is analyzed from multiple levels and dimensions, avoiding the limitations of single feature comparison of traditional methods, and being able to more comprehensively understand and capture the potential similarity of the text. The text features are fused with the semantic features to generate mimetic features, and then the similarity of the duplicate checking is accurately calculated by comparing with the historical database, thereby achieving more efficient duplicate checking detection. When the similarity of the duplicate checking is lower than the set threshold, the graph neural network is further introduced to mine and compare the text structure features, and the structural differences of the text are dealt with by adjusting the similarity, avoiding the errors of the duplicate checking caused by the structural differences. The accuracy of the duplicate checking is improved, the similarity of the target text is comprehensively evaluated, and the precision and robustness of the duplicate checking are improved.

[0058] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0059] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0060] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An intelligent duplicate detection method based on multi-feature mimicry of target entities, characterized in that: include: Collecting target entity data to be detected, and performing word segmentation on the target entity data to be detected based on natural language processing; The data after word segmentation is processed based on the bag-of-words model to extract text features, and the data after word segmentation is processed based on BERT to extract semantic features; The text features and semantic features are integrated to obtain mimetic features, and the mimetic features are compared with the historical database to determine the duplicate check files and determine the duplicate check similarity; The duplicate check similarity is compared with a similarity threshold, and whether to adjust the duplicate check similarity is determined according to the comparison result; when the duplicate check similarity is less than the similarity threshold, it is determined to adjust the duplicate check similarity, and the data after word segmentation is processed based on a graph neural network to extract structural features, and the structural features are compared with the historical structural features in the historical database, and an adjustment coefficient is determined according to the comparison result to adjust the duplicate check similarity.

2. The intelligent duplicate detection method based on target entity multi-feature mimicry according to claim 1 is characterized in that: When the target entity data to be detected is segmented based on natural language processing, it includes: Performing data cleaning on the target entity data to be detected, wherein the data cleaning includes removing stop words and punctuation marks and converting English text into lowercase letters; Perform word count on the cleaned data, including the number of Chinese characters and the number of English characters, and determine the word segmentation tool according to the ratio of the number of Chinese characters to the number of English characters; When the ratio of the number of Chinese characters to the number of English characters is greater than 1, the word segmentation tools include Jieba, THULAC and HanLP; When the ratio of the number of Chinese characters to the number of English characters is less than or equal to 1, the word segmentation tools include NLTK, spaCy and StanfordNLP.

3. The intelligent duplicate detection method based on target entity multi-feature mimicry according to claim 2 is characterized in that: When processing the segmented data based on the bag-of-words model to extract text features, it includes: Pre-build a vocabulary V={w1,w2,...,wn}, where Wn represents the nth word in the vocabulary; Calculate the number of occurrences of each word in the word segmented data in the vocabulary; All occurrence times of the segmented data are represented by feature vectors to obtain the text features.

4. The intelligent duplicate detection method based on target entity multi-feature mimicry according to claim 3 is characterized in that: When the text features and semantic features are integrated to obtain mimicry features, it includes: The text features and the semantic features are adjusted to the same dimension by using a zero-filling method; The text features and semantic features are fused by weighted fusion to obtain mimicry features; ； Among them, V represents the mimicry feature, Vw represents the text feature, Vy represents the semantic feature, a1 and a2 represent the weight coefficients, and a1+a2=1.

5. The intelligent duplicate detection method based on target entity multi-feature mimicry according to claim 4 is characterized in that: When comparing the mimicry feature with the historical database to determine the duplicate check file and determine the duplicate check similarity, it includes: The historical database includes historical mimic features of several historical detection target entity data; Calculating the similarity between the mimicry feature and each historical mimicry feature; ； Among them, Si represents the similarity between the mimicry feature and the i-th historical mimicry feature, V represents the mimicry feature, Vsi represents the i-th historical mimicry feature, Represents the magnitude of a vector.

6. The intelligent duplicate detection method based on target entity multi-feature mimicry according to claim 5 is characterized in that: When judging whether to adjust the similarity of the duplicate check according to the comparison result, it includes: When the similarity of the duplicate check is greater than or equal to the similarity threshold, it is determined that the similarity of the duplicate check is not to be adjusted, and the duplicate check file and the similarity of the duplicate check are output; When the duplicate checking similarity is less than the similarity threshold, it is determined to adjust the duplicate checking similarity.

7. The intelligent duplicate detection method based on target entity multi-feature mimicry according to claim 6 is characterized in that: Determining to adjust the similarity of the duplicate check, and processing the segmented data based on the graph neural network to extract structural features, includes: Convert the segmented data into a graph structure, wherein nodes in the graph structure represent basic elements, including words or sentences, and edges in the graph structure represent relationships between elements; Use Word2Vec to map each of the basic elements to a vector of fixed dimension; Update the representation of nodes in each layer based on convolution operations; ； in, represents the representation of node v at the k+1th layer; Represents the set of neighbor nodes of node v; represents the weight matrix of the kth layer; represents the bias term, represents the activation function, Indicates the representation of node u at layer k; The structural features are obtained by performing average pooling on the representations of all nodes.

8. The intelligent duplicate detection method based on target entity multi-feature mimicry according to claim 7 is characterized in that: When the adjustment coefficient is determined according to the comparison result to adjust the similarity of the duplicate check, it includes: Selecting the historical mimicry feature corresponding to the maximum value of the similarity between the mimicry feature and the historical mimicry feature, and obtaining the historical structural feature of the historical mimicry feature, obtaining the modulus difference between the structural feature and the historical structural feature, and determining an adjustment coefficient according to the modulus difference to adjust the similarity; The modulus difference is compared with a first preset modulus difference and a second preset modulus difference respectively, and an adjustment coefficient is determined according to the comparison result to adjust the duplicate similarity; the first preset modulus difference is smaller than the second preset modulus difference; When the modulus difference is less than or equal to a first preset modulus difference, a first adjustment coefficient is determined to adjust the duplicate similarity; when the modulus difference is greater than the first preset modulus difference and less than or equal to a second preset modulus difference, a second adjustment coefficient is determined to adjust the duplicate similarity; when the modulus difference is greater than the second preset modulus difference, a third adjustment coefficient is determined to adjust the duplicate similarity; the first adjustment coefficient is greater than the second adjustment coefficient, the second adjustment coefficient is greater than the third adjustment coefficient, and the adjustment coefficient takes a value range of (0, 1), and the adjusted duplicate similarity is the product of the duplicate similarity and the adjustment coefficient.

9. An intelligent duplicate detection system based on target entity multi-feature mimicry, used for applying the intelligent duplicate detection method based on target entity multi-feature mimicry as described in any one of claims 1 to 8, characterized in that: include: A collection unit is configured to collect target entity data to be detected, and perform word segmentation on the target entity data to be detected based on natural language processing; A processing unit is configured to process the segmented data based on a bag-of-words model to extract text features, and to process the segmented data based on BERT to extract semantic features; A judgment unit is configured to fuse the text features and the semantic features to obtain a mimetic feature, compare the mimetic feature with a historical database, determine a duplicate check file, and determine a duplicate check similarity; The adjustment unit is configured to compare the duplicate check similarity with a similarity threshold, and determine whether to adjust the duplicate check similarity according to the comparison result; when the duplicate check similarity is less than the similarity threshold, determine to adjust the duplicate check similarity, and process the segmented data based on a graph neural network to extract structural features, compare the structural features with the historical structural features in the historical database, and determine an adjustment coefficient according to the comparison result to adjust the duplicate check similarity.

10. The intelligent duplicate detection system based on target entity multi-feature mimicry according to claim 9 is characterized in that: When the adjustment unit determines the adjustment coefficient according to the comparison result to adjust the similarity of the duplicate check, it also includes: The adjustment unit selects the historical mimicry feature corresponding to the maximum value of the duplicate similarity between the mimicry feature and the historical mimicry feature, obtains the historical structural feature of the historical mimicry feature, obtains the modulus difference between the structural feature and the historical structural feature, and determines the adjustment coefficient according to the modulus difference to adjust the duplicate similarity.

Images