Text duplicate checking method and device, equipment, storage medium and computer program product

Through the method of constructing and input word vector transformation neural network, the problem of low accuracy of text plagiarism check in the prior art is solved, semantic-based text plagiarism check is realized, and the accuracy of plagiarism check is improved.

CN119005162AActive Publication Date: 2024-11-22CHINA MOBILE INFORMATION TECHNOLOGY CO LTD +1
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202410875350.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-01
Publication Date
2024-11-22
Estimated Expiration
2044-07-01

AI Technical Summary

Technical Problem

When the existing text mutation checking method based on natural language processing determines whether the text characters are the same, it is difficult to accurately identify text with the same semantics but different characters, resulting in the impact of the accuracy of the mutation checking result.

Method used

By constructing the text matrix to be reliably checked and inputting the preset word vector transformation neural network, reconstructing it using the semantic information of the training text, and generating a vector representation reflecting the text semantic information, thereby realizing semantic-based text reliance.

Benefits of technology

It improves the accuracy of text plagiarism checking, can capture semantic similarities between texts more accurately, and reduces misjudgment and omissions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005162A_ABST
    Figure CN119005162A_ABST
Patent Text Reader

Abstract

The invention discloses a text duplicate checking method and device, equipment, a storage medium and a computer program product, and relates to the technical field of text processing, and the text duplicate checking method comprises the steps that a to-be-checked duplicate text is acquired, and a to-be-checked duplicate text matrix corresponding to the to-be-checked duplicate text is constructed; inputting the to-be-duplicated text matrix into a preset word vector conversion neural network to obtain a to-be-duplicated text vector; obtaining a comparison text vector of each comparison text; and determining the text vector similarity between the text vector to be subjected to duplicate checking and each comparison text vector, and determining a duplicate checking result of the text to be subjected to duplicate checking according to the text vector similarity. According to the method, the word vector conversion neural network can learn different semantic expression modes of the same semantic information in the training process, so that the semantic information is introduced into the text vector, semantic-based text duplicate checking can be realized, and the accuracy of text duplicate checking is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of text processing technology, and in particular to a text duplication checking method, apparatus, device, storage medium and computer program product. Background Art

[0002] At present, in the traditional text duplication detection method based on natural language processing, duplicate content is mainly identified by judging whether the text characters are the same. However, due to the complexity of the language itself, the same text character may have multiple meanings, the same character may convey different semantics, and different characters may have the same semantics, such as synonyms and antonyms. If the duplicate detection is performed only based on whether the text characters are the same, texts with the same semantics but different characters may be omitted, and texts with the same characters but different semantics may be misjudged as duplicates, which affects the accuracy of the text duplication detection results.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a text duplication checking method, device, equipment, storage medium and computer program product, aiming to solve the technical problem of low accuracy of text duplication checking results.

[0005] To achieve the above purpose, the present application proposes a text duplication checking method, wherein the method comprises:

[0006] Obtain the text to be checked for duplicates, and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates;

[0007] Input the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain the to-be-checked duplicate text vector, wherein the word vector conversion neural network is obtained by taking a first text matrix corresponding to a training text as input data and a first text vector as a label for training, the first text vector is obtained by reducing the dimension of a special text matrix, the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text;

[0008] Obtaining a comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the word vector conversion neural network;

[0009] Determine the text vector similarity between the text vector to be checked for duplicate content and each of the comparison text vectors, and determine the duplicate checking result of the text to be checked for duplicate content based on the text vector similarity.

[0010] In one embodiment, the method further comprises:

[0011] Dividing the training text into a plurality of first sentence character strings according to a preset text division rule, and dividing the document text into a plurality of second sentence character strings according to the preset text division rule;

[0012] Constructing first text matrices corresponding to each of the first sentence character strings, and decomposing each of the first text matrices into a first dictionary matrix and a first sparse matrix, and constructing second text matrices corresponding to each of the second sentence character strings, and decomposing each of the second text matrices into a second dictionary matrix and a second sparse matrix;

[0013] For any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix through each of the second sparse matrices to obtain a feature sparse matrix;

[0014] A special text matrix is ​​constructed using the first dictionary matrix corresponding to the target sparse matrix and the feature sparse matrix, and a dimensionality reduction process is performed on the special text matrix to obtain a reduced-dimensional text vector;

[0015] The first text matrix corresponding to the target sparse matrix is ​​used as input data, and the dimension-reduced text vector is used as a training label to train the word vector conversion neural network to obtain a preset word vector conversion neural network.

[0016] In one embodiment, the step of reconstructing the target sparse matrix through each of the second sparse matrices to obtain a feature sparse matrix corresponding to the target sparse matrix includes:

[0017] Determining matrix similarities between the target sparse matrix and each of the second sparse matrices;

[0018] Determine the number of second sparse matrices whose matrix similarity is greater than a second preset threshold in each of the second sparse matrices, and use the number of matrices as the eigenvalue corresponding to the target sparse matrix;

[0019] A feature vector is constructed based on the eigenvalues ​​corresponding to each of the first sparse matrices, and the target sparse matrix is ​​reconstructed based on the eigenvector to obtain a feature sparse matrix.

[0020] In one embodiment, the step of determining the matrix similarity between the target sparse matrix and each of the second sparse matrices comprises:

[0021] For any second sparse matrix in each of the second sparse matrices, determining a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix;

[0022] Determine the first information proportion of the first column vector in the first text matrix corresponding to the target sparse matrix, and determine the second information proportion of the second column vector in the second text matrix corresponding to the second sparse matrix, wherein the first column vector is a column vector of the target sparse matrix, and the second column vector is a column vector of the second sparse matrix;

[0023] Determining an information factor based on a ratio between the first information ratio and the second information ratio;

[0024] determining a column vector similarity between the first column vector and the second column vector;

[0025] The matrix distance is weighted based on the information factor and the column vector similarity to obtain the matrix similarity between the target sparse matrix and the second sparse matrix.

[0026] In one embodiment, the step of determining a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix comprises:

[0027] Divide the first dictionary matrix corresponding to the target sparse matrix into a plurality of first regional matrices based on a preset matrix partitioning rule, and divide the second dictionary matrix corresponding to the second sparse matrix into a plurality of second regional matrices based on the preset matrix partitioning rule;

[0028] For any target area matrix in each of the first area matrices, determining a correlation coefficient between the target area matrix and each of the second area matrices;

[0029] Determine a variation coefficient value corresponding to a coefficient sequence composed of correlation coefficients corresponding to each of the first regional matrices, and determine a distance weight of each of the first regional matrices based on the variation coefficient value, wherein the variation coefficient value is negatively correlated with the distance weight;

[0030] The distance weight is used as the weight of the weighted Euclidean distance to calculate the matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix.

[0031] In one embodiment, the step of constructing a duplicate text matrix corresponding to the duplicate text to be checked includes:

[0032] Performing word segmentation processing on the text to be checked for duplicates to obtain multiple word character strings;

[0033] Marking each character in the text to be checked for duplicates to obtain a character identifier for each character, wherein the character identifiers of identical characters in the characters are identical;

[0034] The number of the multiple word strings is used as the number of rows, the number of characters of the word string with the largest number of characters in the multiple word strings is used as the number of columns, and the character identifier of each character in the multiple word strings is used as an element to construct a text matrix to be checked for duplicates.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a text duplication checking device, which comprises:

[0036] A construction module is used to obtain the text to be checked for duplicates and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates;

[0037] A word vector conversion module is used to input the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain the to-be-checked duplicate text vector, wherein the word vector conversion neural network is obtained by taking a first text matrix corresponding to a training text as input data and a first text vector as a label for training, the first text vector is obtained by reducing the dimension of a special text matrix, the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text;

[0038] An acquisition module, used for acquiring a comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the preset word vector conversion neural network;

[0039] The comparison module is used to determine the text vector similarity between the text vector to be checked for duplicate content and each of the comparison text vectors, and determine the duplicate checking result of the text to be checked for duplicate content according to the text vector similarity.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a text duplication checking device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the text duplication checking method as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the text duplication checking method described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the text duplication checking method as described above.

[0043] In the present application, a text to be checked for duplicates is obtained, and a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates is constructed; the text matrix to be checked for duplicates is input into a preset word vector conversion neural network to obtain a text vector to be checked for duplicates, wherein the word vector conversion neural network is obtained by taking a first text matrix corresponding to a training text as input data and training with a first text vector as a label, the first text vector is obtained by dimensionality reduction of a special text matrix, the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text; a comparison text vector of each comparison text is obtained, wherein the comparison text vector is obtained by inputting the comparison text into a preset word vector conversion neural network; the text vector similarity between the text vector to be checked for duplicates and each comparison text vector is determined, and the duplicate checking result of the text to be checked for duplicates is determined based on the text vector similarity.

[0044] In the present application, the first text matrix of the training text is reconstructed by using a document text with the same semantic information as the training text, and the content of the document text is introduced into the first text matrix to enrich the semantic expression of the first text matrix, so that the word vector conversion neural network can learn different semantic expressions of the same semantic information during the training process, and the word vector conversion neural network can learn people's understanding of the semantic information of the text, so that it can more accurately capture the semantic similarity between texts in subsequent word vector conversions.

[0045] Therefore, when the text matrix to be checked for duplicate content is input into the word vector conversion neural network for conversion, the word vector conversion neural network can generate a vector representation that can reflect the semantic information of the text. Compared with comparing text characters, the present application introduces semantic information into the text vector to be checked for duplicate content, which can realize semantic-based text checking for duplicate content and improve the accuracy of text checking for duplicate content. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A flowchart of the first embodiment of the method for checking duplicate text in this application is provided;

[0049] Figure 2 A flowchart diagram of the second embodiment of the method for checking duplicate text in this application;

[0050] Figure 3 A brief flowchart of a text duplication checking method provided in one embodiment of the present application;

[0051] Figure 4 This is a schematic diagram of the module structure of the text duplication checking device according to an embodiment of the present application;

[0052] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the text duplication checking method in the embodiment of the present application.

[0053] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0055] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0056] The main solution of the embodiment of the present application is: obtain the text to be checked for duplicates, and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates; input the text matrix to be checked for duplicates into a preset word vector conversion neural network to obtain a text vector to be checked for duplicates; obtain a comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the word vector conversion neural network; determine the text vector similarity between the text vector to be checked for duplicates and each of the comparison text vectors, and determine the duplication checking result of the text to be checked for duplicates based on the text vector similarity.

[0057] In this embodiment, for ease of description, the following description is made using a text duplication checking device as the execution subject.

[0058] As a method of text similarity detection, intelligent text duplicate checking has been widely used. The significance of intelligent text duplicate checking is to find two or more texts or articles from different sources and compare the similarities between them. This comparison is usually done by calculating the distance and similarity between texts to determine the duplication phenomenon between texts. In the process of text duplicate checking, the traditional duplicate checking method is to judge whether it is the same text. However, due to the existence of synonyms, near-synonyms and other information, that is, the corresponding texts are different but represent the same semantic information. If it is judged only based on whether the text is the same or not, there will be a lot of duplicate information that is not found. When the text includes many identical characters, but the semantic information represented is completely different, the same wrong duplicate checking results will be obtained. In the traditional text duplicate checking method based on natural language processing, the text information is directly converted into a text vector according to the number of identical characters, and the similarity between the two text vectors is calculated to obtain the duplicate checking result. Since the semantic information expressed by the text data is not taken into account, the text duplicate checking effect is not ideal and the accuracy is low.

[0059] The embodiment of the present application reconstructs the first text matrix of the training text by using the document text with the same semantic information as the training text, introduces the content of the document text into the first text matrix, enriches the semantic expression of the first text matrix, and enables the word vector conversion neural network to learn different semantic expressions of the same semantic information during the training process, and enables the word vector conversion neural network to learn people's understanding of the semantic information of the text, so that it can more accurately capture the semantic similarity between texts in the subsequent word vector conversion. Thus, when the matrix of the text to be checked for duplicates is input into the word vector conversion neural network for conversion, the word vector conversion neural network can generate a vector representation that can reflect the semantic information of the text. Compared with comparing text characters, the present application introduces semantic information into the text vector to be checked for duplicates, which can realize semantic-based text duplication checking and improve the accuracy of text duplication checking.

[0060] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a text duplicate checking device, etc. The following takes a text duplicate checking device as an example to illustrate this embodiment and the following embodiments.

[0061] Based on this, the present application embodiment provides a text duplication checking method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for checking for duplicate text in this application.

[0062] In this embodiment, the text duplication checking method includes steps S10 to S40.

[0063] Step S10, obtaining the text to be checked for duplicates, and constructing a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates.

[0064] It should be noted that the text that needs to be checked for duplicate content is referred to as the text to be checked for duplicate content, and the two-dimensional matrix corresponding to the text to be checked for duplicate content is referred to as the text matrix to be checked for duplicate content. The text to be checked for duplicate content is converted into a text matrix in numerical form for subsequent machine learning and data analysis. The method of constructing the text matrix to be checked for duplicate content is not limited here and can be set according to actual needs.

[0065] For example, in one feasible implementation, the process of constructing a matrix of texts to be checked for duplicates may be: preprocessing the texts to be checked for duplicates to obtain a plurality of standard words, which may specifically include preprocessing operations such as word segmentation, removal of stop words, stem extraction, and word form restoration; inputting each standard word into a bag-of-words model to obtain a text vector of each standard word; arranging each text vector according to the order of each standard word in the texts to be checked for duplicates to obtain a text matrix.

[0066] For example, in another feasible implementation, the process of constructing the matrix of texts to be checked for duplicates may be:

[0067] Step S101, performing word segmentation processing on the text to be checked for duplicate content to obtain multiple word character strings.

[0068] In this embodiment, there is no restriction on the word segmentation processing method, which can be set according to actual needs. In one feasible implementation, a dictionary-based word segmentation method can be adopted, such as word segmentation based on forward maximum matching method (Forward Maximum Matching, FMM) and word segmentation based on backward maximum matching method (Backward Maximum Matching, BMM); in another feasible implementation, word segmentation tools can also be used for word segmentation, such as Jieba word segmentation, LAC (Lexical Analysis of Chinese, a lexical analysis tool launched by Baidu), THULAC (Tsinghua University Lexical Analyzer for Chinese, a Chinese lexical analysis tool developed and launched by the Natural Language Processing and Social Humanities Computing Laboratory of Tsinghua University), LTP (Language Technology Platform, a Chinese natural language processing platform developed by the Social Computing and Information Retrieval Research Center of Harbin Institute of Technology), etc.

[0069] Step S102, marking each character in the text to be checked for duplicates, and obtaining a character identification of each character, wherein the character identifications of identical characters in each character are identical.

[0070] It should be noted that the character identifier is a unique identifier used to characterize the identity of the character. The same character has the same character identifier. The character identifier is a numerical identifier that provides a basis for the subsequent construction of a numerical text matrix. The character identifier provides a unique identifier for each character, which helps to accurately track and reference each character in the subsequent text matrix construction process, so that each element (i.e., character) in the text matrix can be identified and distinguished, ensuring that there will be no confusion or errors when constructing the text matrix, which helps to improve the accuracy of text processing and analysis.

[0071] Step S103, constructing a matrix of texts to be checked for duplicate content, using the number of word strings as the number of rows, the number of characters in the word string with the largest number of characters as the number of columns, and the character identifier of each character in the word strings as an element.

[0072] It should be noted that since the number of characters in each word string may be inconsistent, there may be a situation where the number of elements in the row data is less than the number of columns. You can add 0 after the row data to ensure that all word strings have the same length, so that the data format can be kept consistent in subsequent data processing and analysis, which is convenient for subsequent calculations and processing.

[0073] Step S20, input the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain the to-be-checked duplicate text vector, wherein the word vector conversion neural network is obtained by taking the first text matrix corresponding to the training text as input data and the first text vector as a label for training, the first text vector is obtained by dimensionality reduction of the special text matrix, the special text matrix is ​​obtained by reconstructing the first text matrix based on the document text, and the document text has the same semantic information as the training text.

[0074] It should be noted that the training text is the text used for model training, the text matrix of the training text is called the first text matrix, the text with the same semantic information as the training text is called the document text, and the document text can be manually selected. Before using the training text to train the model, the first text matrix is ​​reconstructed by the document text, the content of the document text is introduced into the first text matrix to obtain a special text matrix, the special text matrix includes different semantic expressions of the same semantics, and the special text matrix is ​​reduced in dimension to obtain the first text vector, and the first text vector is used for model training. The first text matrix is ​​used as input data, and the first text vector is used as a label training to obtain a word vector conversion neural network. The structure of the word vector conversion neural network is not limited here, and can be set according to actual needs. For example, it can be a neural network based on Word2Vec (Word to Vector, word vector model), a neural network based on RNN (Recurrent Neural Network, recurrent neural network) or LSTM (Long Short-Term Memory, long short-term memory network), a neural network based on DNN (Deep Neural Networks, deep neural network), etc. The matrix of text to be checked for duplicate content is input into the trained word vector conversion neural network. The obtained vector is called the vector of text to be checked for duplicate content. The vector of text to be checked for duplicate content is used for subsequent text checking for duplicate content.

[0075] Step S30, obtaining a comparison text vector of each comparison text, wherein the comparison text vector is obtained by converting the comparison text input word vector into a neural network.

[0076] It should be noted that the text to be compared with the text to be checked for plagiarism is called comparison text, and the vector of the comparison text is called comparison text vector.

[0077] The comparison text vector is obtained by converting the comparison text input word vector into a neural network. In a feasible implementation, before the text is checked for plagiarism, the comparison text input word vector is converted into a neural network to obtain the comparison text vector, the comparison text vector is stored in a database, and the comparison text vector is obtained from the database during the text check for plagiarism. In the scenario where the comparison text is a fixed text, this implementation can reduce the text turning amount operation during the text check for plagiarism and improve the efficiency of the text check for plagiarism. For example, in the scenario of paper comparison and plagiarism, the comparison texts of different papers are all existing documents in the paper library and the document library. The existing documents in the paper library and the document library are converted into comparison text vectors in advance, which can improve the efficiency of text check for plagiarism and reduce the consumption of computing resources. In another feasible implementation, the comparison text input word vector is converted into a neural network to obtain the comparison text vector during the text check for plagiarism. It can be specifically set according to actual needs and is not limited here.

[0078] Step S40, determining the text vector similarity between the text vector to be checked for duplicate content and each comparison text vector, and determining the duplicate checking result of the text to be checked for duplicate content according to the text vector similarity.

[0079] It should be noted that the method of determining the text vector similarity between the text vector to be checked for duplicates and each comparison text vector is not limited here and can be set according to actual needs. For example, the cosine similarity, Jaccard similarity, Euclidean distance or Manhattan distance between the text vector to be checked for duplicates and each comparison text vector can be used as the text vector similarity.

[0080] The duplicate checking result of the text to be checked is determined according to the text vector similarity, wherein the text vector similarity represents the similarity between the text to be checked for duplicates and the comparison text, that is, the higher the vector similarity, the higher the similarity between the text to be checked for duplicates and the comparison text.

[0081] In this embodiment, the first text matrix of the training text is reconstructed by using a document text with the same semantic information as the training text, and the content of the document text is introduced into the first text matrix to enrich the semantic expression of the first text matrix, so that the word vector conversion neural network can learn different semantic expressions of the same semantic information during the training process, and the word vector conversion neural network can learn people's understanding of the semantic information of the text, so that it can more accurately capture the semantic similarity between texts in subsequent word vector conversions.

[0082] Therefore, when the matrix of the text to be checked for duplicate content is input into the word vector conversion neural network for conversion, the word vector conversion neural network can generate a vector representation that can reflect the semantic information of the text. Compared with comparing text characters, this embodiment introduces semantic information into the vector of the text to be checked for duplicate content, which can realize semantic-based text checking for duplicate content and improve the accuracy of text checking for duplicate content.

[0083] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can refer to the above introduction, and will not be repeated later. Figure 2 The text duplication checking method also includes steps S50 to S90.

[0084] Step S50, dividing the training text into a plurality of first sentence character strings according to a preset text division rule, and dividing the document text into a plurality of second sentence character strings according to a preset text division rule.

[0085] It should be noted that the preset text segmentation rules can be set according to actual needs and are not limited here. For example, in a feasible implementation, the text can be segmented with punctuation marks as nodes, and the text characters between two adjacent punctuation marks can be divided into a sentence string. For example, the text characters between two adjacent sentence-end punctuation marks (such as a period, exclamation mark, greeting, ellipsis, etc., which indicate the end of a sentence) can be divided into a sentence string. Compared with the sentence string segmentation with the pauses and structural relationships within the sentence (such as semicolons, commas, etc.) as nodes, this implementation can make the sentence string obtained by the segmentation have complete semantic information, so that the word vector neural network can learn the complete semantic information of the context composition during the training process, enrich the semantic expression of the word vector, and improve the accuracy of the subsequent use of the word vector for text duplication checking.

[0086] Step S60, constructing a first text matrix corresponding to each first sentence character string, and decomposing each first text matrix into a first dictionary matrix and a first sparse matrix, and constructing a second text matrix corresponding to each second sentence character string, and decomposing each second text matrix into a second dictionary matrix and a second sparse matrix.

[0087] It should be noted that the method of constructing the first text matrix is ​​not limited here, and the details may refer to step S10.

[0088] For each first text matrix, the first text matrix is ​​decomposed into a first dictionary matrix and a first sparse matrix to reduce the dimension of the first text data while retaining important information.

[0089] The first dictionary matrix represents a set of basis vectors in the first text matrix, and the basis vectors can capture the common text information in the first text matrix. The first sparse matrix represents the unique text information in the first text matrix. Since the probability of different sentence strings containing the same characters is low, the similarity between the dictionary matrices corresponding to different sentence strings is low. When the semantic information corresponding to different sentence strings is similar, there is similarity between the corresponding sparse matrices. Therefore, this embodiment obtains the reconstructed text matrix according to the changes of the dictionary matrix and the sparse matrix corresponding to the text matrix, and further obtains the semantic information contained in the text matrix. In a specific implementation, there is no limitation on the way of decomposing the first text matrix. For example, it can be an SVD (Singular Value Decomposition) algorithm, a K-SVD (K-Singular Value Decomposition) algorithm, etc. Similarly, the same operation as the first sentence string is performed on each second sentence string, which will not be repeated here.

[0090] Step S70: for any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix through each of the second sparse matrices to obtain a feature sparse matrix.

[0091] It should be noted that the target sparse matrix is ​​any one of the first sparse matrices. Since the sparse matrix represents the semantic information of the text matrix, the target sparse matrix can be reconstructed using the second sparse matrix to enrich the expression form of the target sparse matrix for the same semantics.

[0092] The manner of reconstructing the target sparse matrix by using the second sparse matrices is not limited here.

[0093] For example, in a feasible implementation, the process of reconstructing the target sparse matrix through each second sparse matrix can be: for each second sparse matrix, based on a sparse coding algorithm (such as Lasso (Least Absolute Shrinkage and Selection Operator), OMP (Orthogonal Matching Pursuit), etc.), the column vectors in the second sparse matrix are processed to obtain the sparse coding coefficients of each column vector; using the column vectors in the second sparse matrix and the respective sparse coding coefficients, the target sparse matrix is ​​reconstructed in a linear combination manner.

[0094] For example, in another feasible implementation manner, the process of reconstructing the target sparse matrix through each second sparse matrix may include steps S701 to S703.

[0095] Step S701, determining the matrix similarity between the target sparse matrix and each second sparse matrix.

[0096] It should be noted that matrix similarity is used to characterize the similarity between the first text matrix corresponding to the target sparse matrix and each second text matrix at the character level. Matrix similarity can reveal the closeness between texts in terms of character sequence, spelling, word composition, etc. For example, if two text matrices contain a large number of identical character sequences or words, their matrix similarity may be very high. The method of determining matrix similarity is not limited here. For example, the length of the LCS (Length of Longest Common Subsequence) between the target sparse matrix and the second coefficient matrix, Jaccard similarity, or similarity based on edit distance can be used as matrix similarity.

[0097] In a feasible implementation, the process of determining matrix similarity may include A1 to A5.

[0098] A1. For any second sparse matrix among the second sparse matrices, determine a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix.

[0099] It should be noted that the following same processing is performed on any second sparse matrix among the second sparse matrices, and the following description is given by taking any second sparse matrix as an example.

[0100] The matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix represents the degree of difference between the two dictionary matrices in structure or content. A smaller matrix distance means that the two dictionary matrices are more similar, that is, the text data they represent are closer in structure or content; while a larger matrix distance indicates that the two dictionary matrices are more different, and the text data they represent are quite different in structure or content.

[0101] The method of determining the matrix distance is not limited here. For example, the Euclidean distance and Frobenius norm (Frobenius norm) between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix may be calculated as the matrix distance.

[0102] Specifically, in a feasible implementation manner, the process of determining the matrix distance may include A11 to A13.

[0103] A11, divide the first dictionary matrix corresponding to the target sparse matrix into multiple first regional matrices based on a preset matrix division rule, and divide the second dictionary matrix corresponding to the second sparse matrix into multiple second regional matrices based on a preset matrix division rule.

[0104] It should be noted that the preset matrix division rule can be set according to actual needs and is not limited here.

[0105] A12: for any target region matrix in each of the first region matrices, determine a correlation coefficient between the target region matrix and each of the second region matrices.

[0106] It should be noted that the correlation coefficient represents the linear correlation strength between the target region matrix and each second region matrix, and may specifically be a Pearson correlation coefficient.

[0107] A13, determining a coefficient of variation value corresponding to a coefficient sequence composed of correlation coefficients corresponding to each first regional matrix, and determining a distance weight of each first regional matrix based on the coefficient of variation value, wherein the coefficient of variation value is negatively correlated with the distance weight.

[0108] It should be noted that the smaller the coefficient of variation value is, the more uniform the target region matrix is ​​compared with the second region matrix, the more similarity information the target region matrix and the second region matrix contain, and the weights corresponding to the matrix elements of the target region matrix should be set larger for subsequent calculation of the weighted Euclidean distance.

[0109] A14, using the distance weight as the weight of the weighted Euclidean distance, and calculating the matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix.

[0110] Specifically, the calculation formula of the weighted Euclidean distance can be:

[0111]

[0112] Where D(X, Y) is the weighted Euclidean distance between the first dictionary matrix and the second dictionary matrix, X represents the first dictionary matrix, Y represents the second dictionary matrix, and X i represents the i-th first region matrix among the first region matrices obtained by dividing the first dictionary matrix, Y i represents the i-th second region matrix among the second region matrices obtained by partitioning the second dictionary matrix, ω i Represents the weight of the i-th first region matrix.

[0113] A2, determine the proportion of first information of the first column vector in the first text matrix corresponding to the target sparse matrix, and determine the proportion of second information of the second column vector in the second text matrix corresponding to the second coefficient matrix, wherein the first column vector is the column vector of the target sparse matrix, and the second column vector is the column vector of the second sparse matrix.

[0114] It should be noted that the information ratio represents the relative size of the information amount or importance carried by the column vector in the text matrix, the first information ratio represents the relative size of the information amount or importance carried by the first column vector in the first text matrix corresponding to the target sparse matrix, and the second information ratio represents the relative size of the information amount or importance carried by the second column vector in the second text matrix corresponding to the second coefficient matrix.

[0115] The specific method for obtaining the first information ratio of the first column vector is: setting the first column vector to 0, and using the target sparse matrix after the first column vector is set to 0 as the new sparse matrix; reconstructing the new sparse matrix and the first dictionary matrix of the target coefficient matrix to obtain a new text matrix; calculating the inverse function of the Euclidean distance between the new text matrix and the first text matrix corresponding to the target coefficient matrix as the information ratio of the first column vector.

[0116] The information ratio of the second column vector can refer to the calculation process of the first column vector, which will not be repeated here.

[0117] A3, determining the information factor based on the ratio between the first information ratio and the second information ratio.

[0118] It should be noted that the information factor represents the relative importance between the first information ratio and the second information ratio, and is used to perform weighted processing on the matrix distance in the subsequent process to obtain the matrix similarity. The information factor can be the ratio of the first information ratio divided by the second information ratio; or the ratio of the second information ratio divided by the first information ratio, which is not limited here.

[0119] A4, determining the column vector similarity between the first column vector and the second column vector.

[0120] It should be noted that the method of calculating the column vector similarity is not limited here. For example, the cosine similarity, Jaccard similarity, Euclidean distance or Manhattan distance between the first column vector and the second column vector can be used as the column vector similarity.

[0121] A5, weighting the matrix distance based on the information factor and the column vector similarity to obtain the matrix similarity between the target sparse matrix and the second sparse matrix.

[0122] Step S702, determining the number of second sparse matrices whose matrix similarity in each second sparse matrix is ​​greater than a second preset threshold, and using the number of matrices as the eigenvalue corresponding to the target sparse matrix.

[0123] It should be noted that by setting a similarity threshold to extract eigenvalues, it is possible to extract information related to the target sparse matrix from a large number of second sparse matrices and simplify this information into a single eigenvalue, which can reduce the operational complexity in the subsequent sparse matrix reconstruction process and improve the efficiency of matrix reconstruction. It can also filter out sparse matrices with low similarity to the target sparse matrix, thereby reducing the impact of noise and outliers on the subsequent reconstructed sparse matrix, so as to improve the overall quality and accuracy of the subsequent reconstructed sparse matrix.

[0124] Step S703: constructing eigenvectors based on the eigenvalues ​​corresponding to each of the first sparse matrices, and reconstructing the target sparse matrix based on the eigenvectors to obtain a characteristic sparse matrix.

[0125] It should be noted that the feature vector and feature sparse matrix provide richer information than the original sparse matrix, and can better describe the similarities and differences between texts to improve the accuracy of subsequent text duplication checking.

[0126] Step S80, using the first dictionary matrix and the feature sparse matrix corresponding to the target sparse matrix to construct a special text matrix, and performing dimensionality reduction processing on the special text matrix to obtain a reduced-dimensional text vector.

[0127] The feature sparse matrix and the first dictionary matrix are used for inverse decomposition operations to reconstruct the special text matrix, and the special text matrix is ​​subjected to matrix dimensionality reduction to obtain the reduced-dimensional text vector, which converts the high-dimensional data into a low-dimensional space, significantly reducing the dimension of the data. In the subsequent text duplication check process, the cost of computing resources can be reduced, and the efficiency of subsequent text duplication check can be improved. In addition, in the process of dimensionality reduction, some noise and redundant features are usually removed, making the data purer and easier to analyze, so as to improve the accuracy and reliability of subsequent text duplication check.

[0128] It should be noted that there is no restriction on the method of performing matrix dimensionality reduction on the special text matrix. For example, it can be based on principal component analysis (PCA), singular value decomposition (SVD) or truncated singular value decomposition (Truncated SVD), non-negative matrix factorization (NMF), etc. The specific settings can be made according to actual needs.

[0129] Step S90, using the first text matrix corresponding to the target sparse matrix as input data and the dimension-reduced text vector as training label, training the word vector conversion neural network to be trained to obtain a preset word vector conversion neural network.

[0130] In this embodiment, the training text is divided into multiple first sentence character strings according to a preset text division rule, and the document text is divided into multiple second sentence character strings according to a preset text division rule; a first text matrix corresponding to each first sentence character string is constructed, and each first text matrix is ​​decomposed into a first dictionary matrix and a first sparse matrix, and a second text matrix corresponding to each second sentence character string is constructed, and each second text matrix is ​​decomposed into a second dictionary matrix and a second sparse matrix; for any target sparse matrix in each first sparse matrix, the target sparse matrix is ​​reconstructed by each second sparse matrix to obtain a feature sparse matrix; a special text matrix is ​​constructed using the first dictionary matrix and the feature sparse matrix corresponding to the target sparse matrix, and the special text matrix is ​​subjected to dimensionality reduction processing to obtain a reduced dimensionality text vector; the first text matrix corresponding to the target sparse matrix is ​​used as input data, and the reduced dimensionality text vector is used as a training label to train the word vector conversion neural network to obtain a preset word vector conversion neural network.

[0131] In this implementation, the first text matrix of the training text is reconstructed by using a document text with the same semantic information as the training text, and the content of the document text is introduced into the first text matrix to enrich the semantic expression of the first text matrix, so that the word vector conversion neural network can learn different semantic expressions of the same semantic information during the training process, and the word vector conversion neural network can learn people's understanding of the semantic information of the text, so that it can more accurately capture the semantic similarities between texts in subsequent word vector conversions, so as to improve the accuracy of subsequent text duplication checking based on word vectors.

[0132] For example, to help understand the technical concept or technical principle of the text duplication checking method after the present embodiment is combined with the above-mentioned embodiment 1 and embodiment 2, please refer to Figure 3 , Figure 3 A brief flowchart of a text duplication checking method is provided, as follows:

[0133] Step S001: Obtain sample text and document text. The document text has the same semantic information as the training text. Specifically, professionals can obtain the sample text and document text through manual classification. Texts with the same semantic information are texts with the same meaning. Professionals can analyze and judge based on the synonyms or synonyms of existing vocabulary and sentence transformation.

[0134] Step S002: Divide the training text into a plurality of first sentence strings according to a preset text division rule, and construct a first text matrix corresponding to each first sentence string. Divide the document text into a plurality of second sentence strings according to a preset text division rule, and construct a second text matrix corresponding to each second sentence string.

[0135] In step (2.1), the character string between two adjacent periods in the text data is recorded as a sentence character string.

[0136] Step (2.2): Perform Jieba word segmentation on any sentence string to obtain several words in the sentence string, denoted as the word string, in order to construct a two-dimensional matrix, that is, the text matrix. The specific process of constructing the two-dimensional matrix can be as follows: Obtain the maximum value of the number of characters contained in all word strings in the sentence string, denoted as the first value; denote the number of word strings included in the sentence string as the second value; use the first value as the number of columns of the two-dimensional matrix and the second value as the number of rows of the two-dimensional matrix; classify the same characters in the sentence string as one category, obtain the number of types of characters in the sentence string, start marking each type of character from the number 1, denoted as the marking number (i.e., the character identifier); denote the sequence formed by the marking numbers of the characters included in the word string as the row data of the word string; according to the order of the word strings in the sentence string, use the row data of the word strings as the row vectors of the two-dimensional matrix; when the number of columns of the row data is different from the number of columns of the two-dimensional matrix, perform a padding operation of adding 0s at the end; denote the two-dimensional matrix formed by the row data of all word strings as the text matrix of the sentence string, denote the text matrix of the sentence string in the first text as the first text matrix, and denote the text matrix of the sentence string in the second text as the second text matrix. For example, for the sentence string "Construct a text matrix", the included word strings are "Construct" and "text matrix" respectively. Among them, denote the character "构" as 1, the character "建" as 2, the character "文" as 3, the character "本" as 4, the character "矩" as 5, and the character "阵" as 6. Then the row data corresponding to the word string "Construct" is [1, 2], and the row data corresponding to "text matrix" is [3, 4, 5, 6].

[0137] Step S003: Reconstruct the first text matrix to obtain a special text matrix. Specifically, for any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix through each of the second sparse matrices to obtain a feature sparse matrix, and construct a special text matrix using the first dictionary matrix corresponding to the target sparse matrix and the feature sparse matrix.

[0138] Step (3.1), using the K-SVD decomposition algorithm to decompose any text matrix into a corresponding dictionary matrix and a sparse matrix, record the sparse matrix corresponding to the first text matrix as the first sparse matrix, record the sparse matrix corresponding to the second text matrix as the second sparse matrix, record the column vectors in any sparse matrix as sparse column vectors, record the sparse column vectors in the first sparse matrix and the second sparse matrix as the first column vector and the second column vector, respectively, to obtain a plurality of first column vectors and second column vectors. The dictionary matrix represents the common text information in the text matrix, and the sparse matrix represents the unique text information in the text matrix. Since the probability of the same characters contained in different sentence strings is low, the similarity between the dictionary matrices corresponding to different sentence strings is low, but when the semantic information corresponding to different sentence strings is similar, there is similarity between the corresponding sparse matrices. Therefore, this embodiment obtains the reconstructed text matrix according to the changes in the dictionary matrix and the sparse matrix corresponding to the text matrix, and further obtains the semantic information contained in the text matrix. When the semantic information between different sentence strings is close, the sparse matrices corresponding to the two sentence strings are similar. The sparse matrix is ​​reset according to the similarity between the dilution matrices, and then the text matrix is ​​reconstructed to introduce semantic information. It should be noted that by obtaining the similarity between the sparse column vectors in the first sparse matrix and the sparse column vectors in the second sparse matrix, the best matching combination of the column vectors of the sparse matrix is ​​obtained, and this matching combination reflects the best semantic information matching result.

[0139] Step (3.2) constructs a bipartite graph. The method for constructing the bipartite graph is as follows: the first column vector is recorded as the node on the left side of the bipartite graph as the left node, the second column vector is recorded as the node on the right side of the bipartite graph as the right node, and the connection relationship between the left node and the right node in the bipartite graph is called an edge.

[0140] Step (3.3) obtains the similarity between the i-th left node and the j-th right node in the bipartite graph (that is, the matrix similarity between the target sparse matrix and each second sparse matrix). The specific calculation method is:

[0141]

[0142] Among them, α ij represents the similarity between the i-th left node and the j-th right node in the bipartite graph; τ ij represents the difference between the i-th left node and the j-th right node; λ i represents the information proportion of the column vector corresponding to the i-th left node, λ j represents the information proportion of the sparse column vector corresponding to the j-th right node; cos(i, j) represents the cosine similarity between the sparse column vectors corresponding to the i-th left node and the j-th right node.

[0143] It should be noted that the weighted Euclidean distance between the dictionary matrices of the text matrix to which the sparse matrix corresponding to any left node and right node belongs is obtained, which is recorded as the difference between the left node and the right node. When the characters included in the two sentence strings are the same, the two text matrices are the same, and the dictionary matrices after K-SVD decomposition may also have the same information. Therefore, if only the Euclidean distance is used for measurement, there will be a large difference, and the structural information of the dictionary matrix is ​​analyzed accordingly. In this embodiment, the adaptive Euclidean distance is calculated by assigning a weight to each element in the dictionary matrix, wherein the dictionary matrix is ​​divided into eight equal regions, wherein the Pearson correlation coefficient is calculated between a single region of the first text dictionary matrix and each region of the second text matrix, and then a Pearson correlation coefficient sequence of a single region of the first text dictionary matrix can be obtained, and the coefficient of variation of the sequence is calculated. If the coefficient of variation is smaller, it indicates that the region in the first text dictionary matrix is ​​more uniform than that in the second text dictionary matrix, and it may contain more similarity information, then the matrix element of the region in the corresponding first text dictionary matrix should be set larger when obtaining the weight, that is, the matrix element of the region is an inverse proportional function of the coefficient of variation, and then a weighted Euclidean distance is performed. Distance calculation (i.e., dividing the first dictionary matrix corresponding to the target sparse matrix into a plurality of first region matrices based on a preset matrix division rule, and dividing the second dictionary matrix corresponding to the second sparse matrix into a plurality of second region matrices based on a preset matrix division rule; for any target region matrix in each first region matrix, determining the correlation coefficient between the target region matrix and each second region matrix; determining the coefficient of variation corresponding to the coefficient sequence composed of the correlation coefficients corresponding to each first region matrix, and determining the distance weight of each first region matrix based on the coefficient of variation, wherein the coefficient of variation is negatively correlated with the distance weight; using the distance weight as the weight of the weighted Euclidean distance, calculating the matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix). It should be noted that the smaller the difference between the dictionary matrices corresponding to the first text matrix and the second text matrix, the more characters the first text and the second text include, and the more it is necessary to consider the difference between the sparse column vectors corresponding to the first text matrix and the second text matrix, so as to reflect the representation ability of the two sparse column vectors on the semantic information of the text, that is, the size of the cosine similarity between the two sparse column vectors needs to be considered. It should be noted that the specific method for obtaining the information ratio of the column vector is as follows: the sparse column vector of any sparse matrix is ​​set to 0, the sparse matrix after the sparse column vector is set to 0 is recorded as the new sparse matrix of the text matrix, the new sparse matrix of the text matrix and the dictionary matrix are combined, and the K-SVD decomposition algorithm is used to reconstruct the reconstructed text matrix, which is recorded as the new text matrix corresponding to the sparse column vector, and the inverse of the Euclidean distance d between the text matrix of the sparse matrix to which the sparse column vector belongs and the new text matrix is ​​obtained. It is recorded as the information ratio of the column vector. It should be noted that this embodiment uses the cosine similarity between the sparse column vectors to characterize the connection characteristics between two sparse column vectors. The greater the connection between the two sparse column vectors, the more similar the semantic information represented by the two sparse column vectors.

[0144] Step (3.4), using the similarity between the left node and the right node as the edge weight of the edge connecting the left node and the right node in the bipartite graph, obtaining the number of any left node and all right nodes whose edge weights are greater than the preset edge weight threshold, and recording it as the eigenvalue of the first column vector corresponding to the left node (that is, determining the number of matrices of the second sparse matrix whose matrix similarity is greater than the second preset threshold in each second sparse matrix, and using the number of matrices as the eigenvalue corresponding to the target sparse matrix). It should be noted that the edge weight threshold can be preset as 0.65 based on experience, and can be adjusted according to specific circumstances, and this embodiment does not make specific limitations. It should be noted that the size of the eigenvalue of the first column vector represents the number of second column vectors with high similarity in the second sparse matrix, reflecting the similarity between the first column vector and the second sparse matrix. The larger the eigenvalue, the greater the similarity between the first column vector and the second sparse matrix.

[0145] Step (3.5), obtain the eigenvalues ​​of all first column vectors in the first sparse matrix, and perform linear normalization on all eigenvalues ​​to obtain normalized eigenvalues, record the column vectors formed by all normalized eigenvalues ​​of the first sparse matrix as the eigenvectors of the first sparse matrix, multiply the first sparse matrix and the eigenvector to obtain the characteristic sparse matrix of the first text matrix, combine the characteristic sparse matrix and the dictionary matrix of the first text matrix, and reconstruct using the K-SVD decomposition algorithm to obtain the special text matrix of the first text matrix, obtain all special text matrices of the sample text, and one sentence string corresponds to one special text matrix (that is, construct a eigenvector based on the eigenvalues ​​corresponding to each first sparse matrix, reconstruct the target sparse matrix based on the eigenvector, and obtain the characteristic sparse matrix). It should be noted that after manual classification, the special text matrix of the obtained text matrix reflects the specific quantitative features of the text matrix when describing the semantic information of the sentence string corresponding to the text matrix under the same semantic information, which is convenient for subsequent analysis of the similarity between the sentence strings in terms of semantic information.

[0146] Step S004: training the word vector conversion neural network. The specific process is: step (4.1), using the first dictionary matrix and the feature sparse matrix corresponding to the target sparse matrix to construct a special text matrix, and performing dimensionality reduction processing on the special text matrix to obtain a reduced-dimensional text vector. Specifically, the PCA algorithm can be used to reduce the dimension of the special text matrix to a one-dimensional vector to obtain a reduced-dimensional text vector. Step (4.2), using the first text matrix corresponding to the target sparse matrix as input data and the reduced-dimensional text vector as a training label, the word vector conversion neural network to be trained is trained to obtain a preset word vector conversion neural network. It should be noted that the word vector conversion neural network can be a single hidden layer DNN neural network, and the root mean square error function is used as the loss function of the neural network for training.

[0147] Step S005: Use the trained word vector conversion neural network to perform text duplication check. The specific process is: Step (5.1), obtain the text to be checked for duplicates, and construct the text matrix to be checked for duplicates corresponding to the text to be checked for duplicates. Step (5.2), input the text matrix to be checked for duplicates into the trained word vector conversion neural network to obtain the text vector to be checked for duplicates. Step (5.3), obtain the comparison text vectors of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the word vector conversion neural network. Step (5.4), determine the text vector similarity between the text vector to be checked for duplicates and each comparison text vector, and determine the duplicate checking result of the text to be checked for duplicates based on the text vector similarity. It should be noted that when the cosine similarity between the text vector to be checked for duplicates and the comparison text vector is used as the text vector similarity, when the text vector similarity is greater than the preset cosine similarity threshold, the sentence string corresponding to the duplicate checking text vector is marked as repeated. Specifically, the size of the cosine similarity between the duplicate check vector and the document vector reflects the similarity between the corresponding semantic information of the corresponding sentence strings. The larger the cosine similarity, the closer the semantic information expressed by the sentence strings is. The cosine similarity threshold can be preset to 0.78 based on experience and can be adjusted according to actual conditions. This embodiment does not make any specific limitations.

[0148] In this embodiment, the vector conversion of text information is performed by training a word vector conversion neural network, and then the vectors are compared to realize text duplication checking, thereby avoiding the shortcomings of traditional word vector conversion networks that only convert according to word frequency without considering the semantic information of the text. The word vector conversion neural network can describe the text data according to the semantic information of the text data, thereby improving the influence of the semantic information of the text data in the text duplication checking process and further improving the accuracy of text duplication checking.

[0149] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method of checking for duplicate text in the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0150] This application also provides a text duplication checking device, please refer to Figure 4 , the text duplication checking device comprises:

[0151] A construction module 10 is used to obtain a text to be checked for duplicates and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates;

[0152] A word vector conversion module 20 is used to input the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain the to-be-checked duplicate text vector, wherein the word vector conversion neural network is obtained by taking a first text matrix corresponding to a training text as input data and a first text vector as a label for training, the first text vector is obtained by reducing the dimension of a special text matrix, the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text;

[0153] An acquisition module 30, used to acquire a comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the preset word vector conversion neural network;

[0154] The comparison module 40 is used to determine the text vector similarity between the text vector to be checked for duplicate content and each of the comparison text vectors, and determine the duplicate checking result of the text to be checked for duplicate content according to the text vector similarity.

[0155] Optionally, the text duplication checking device further includes a training module, which is used to:

[0156] Dividing the training text into a plurality of first sentence character strings according to a preset text division rule, and dividing the document text into a plurality of second sentence character strings according to the preset text division rule;

[0157] Constructing first text matrices corresponding to each of the first sentence character strings, and decomposing each of the first text matrices into a first dictionary matrix and a first sparse matrix, and constructing second text matrices corresponding to each of the second sentence character strings, and decomposing each of the second text matrices into a second dictionary matrix and a second sparse matrix;

[0158] For any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix through each of the second sparse matrices to obtain a feature sparse matrix;

[0159] A special text matrix is ​​constructed using the first dictionary matrix corresponding to the target sparse matrix and the feature sparse matrix, and a dimensionality reduction process is performed on the special text matrix to obtain a reduced-dimensional text vector;

[0160] The first text matrix corresponding to the target sparse matrix is ​​used as input data, and the dimension-reduced text vector is used as a training label to train the word vector conversion neural network to obtain a preset word vector conversion neural network.

[0161] Optionally, the training module is further used to:

[0162] Determining matrix similarities between the target sparse matrix and each of the second sparse matrices;

[0163] Determine the number of second sparse matrices whose matrix similarity is greater than a second preset threshold in each of the second sparse matrices, and use the number of matrices as the eigenvalue corresponding to the target sparse matrix;

[0164] A feature vector is constructed based on the eigenvalues ​​corresponding to each of the first sparse matrices, and the target sparse matrix is ​​reconstructed based on the eigenvector to obtain a feature sparse matrix.

[0165] Optionally, the training module is further used to:

[0166] For any second sparse matrix in each of the second sparse matrices, determining a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix;

[0167] Determine the first information proportion of the first column vector in the first text matrix corresponding to the target sparse matrix, and determine the second information proportion of the second column vector in the second text matrix corresponding to the second sparse matrix, wherein the first column vector is a column vector of the target sparse matrix, and the second column vector is a column vector of the second sparse matrix;

[0168] Determining an information factor based on a ratio between the first information ratio and the second information ratio;

[0169] determining a column vector similarity between the first column vector and the second column vector;

[0170] The matrix distance is weighted based on the information factor and the column vector similarity to obtain the matrix similarity between the target sparse matrix and the second sparse matrix.

[0171] Optionally, the training module is further used to:

[0172] Divide the first dictionary matrix corresponding to the target sparse matrix into a plurality of first regional matrices based on a preset matrix partitioning rule, and divide the second dictionary matrix corresponding to the second sparse matrix into a plurality of second regional matrices based on the preset matrix partitioning rule;

[0173] For any target area matrix in each of the first area matrices, determining a correlation coefficient between the target area matrix and each of the second area matrices;

[0174] Determine a variation coefficient value corresponding to a coefficient sequence composed of correlation coefficients corresponding to each of the first regional matrices, and determine a distance weight of each of the first regional matrices based on the variation coefficient value, wherein the variation coefficient value is negatively correlated with the distance weight;

[0175] The distance weight is used as the weight of the weighted Euclidean distance to calculate the matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix.

[0176] Optionally, the building module 10 is further used for:

[0177] Performing word segmentation processing on the text to be checked for duplicates to obtain multiple word character strings;

[0178] Marking each character in the text to be checked for duplicates to obtain a character identifier for each character, wherein the character identifiers of identical characters in the characters are identical;

[0179] The number of the multiple word strings is used as the number of rows, the number of characters of the word string with the largest number of characters in the multiple word strings is used as the number of columns, and the character identifier of each character in the multiple word strings is used as an element to construct a text matrix to be checked for duplicates.

[0180] The text duplicate checking device provided by the present application adopts the text duplicate checking method in the above embodiment, which can solve the technical problem of low accuracy of the text duplicate checking result. Compared with the prior art, the beneficial effects of the text duplicate checking device provided by the present application are the same as the beneficial effects of the text duplicate checking method provided by the above embodiment, and the other technical features in the text duplicate checking device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0181] The present application provides a text duplication checking device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the text duplication checking method in the above-mentioned embodiment one.

[0182] Reference below Figure 5, which shows a schematic diagram of the structure of a text duplicate checking device suitable for implementing the embodiment of the present application. The text duplicate checking device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The text duplication checking device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0183] like Figure 5 As shown, the text duplication checking device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the text duplication checking device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the text duplicate checking device to communicate with other devices wirelessly or by wire to exchange data. Although the text duplicate checking device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0184] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0185] The text duplicate checking device provided by the present application adopts the text duplicate checking method in the above embodiment, which can solve the technical problem of low accuracy of the text duplicate checking result. Compared with the prior art, the beneficial effects of the text duplicate checking device provided by the present application are the same as the beneficial effects of the text duplicate checking method provided by the above embodiment, and the other technical features in the text duplicate checking device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0186] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0187] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0188] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the text duplication checking method in the above-mentioned embodiment.

[0189] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0190] The above-mentioned computer-readable storage medium may be included in the text duplication checking device; or it may exist independently without being assembled into the text duplication checking device.

[0191] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the text duplication checking device, the text duplication checking device is enabled to: obtain the text to be checked for duplicates, and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates; input the text matrix to be checked for duplicates into a preset word vector conversion neural network to obtain the text vector to be checked for duplicates, wherein the word vector conversion neural network is obtained by taking the first text matrix corresponding to the training text as input data and the first text vector as a label for training, the first text vector is obtained by reducing the dimension of a special text matrix, the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text; obtain the comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the word vector conversion neural network; determine the text vector similarity between the text vector to be checked for duplicates and each of the comparison text vectors, and determine the duplication checking result of the text to be checked for duplicates according to the text vector similarity.

[0192] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0193] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0194] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0195] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned text duplication checking method, and can solve the technical problem of low accuracy of the results of text duplication checking. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the text duplication checking method provided by the above-mentioned embodiment, and will not be elaborated here.

[0196] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned text duplication checking method when executed by a processor.

[0197] The computer program product provided by the present application can solve the technical problem of low accuracy of the results of text duplication checking. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the text duplication checking method provided by the above embodiment, which will not be repeated here.

[0198] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A text duplication checking method, characterized in that: The method includes: Obtain the text to be checked for duplicates, and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates; Input the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain the to-be-checked duplicate text vector, wherein the word vector conversion neural network is obtained by taking a first text matrix corresponding to a training text as input data and a first text vector as a label for training, the first text vector is obtained by reducing the dimension of a special text matrix, the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text; Obtaining a comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the word vector conversion neural network; Determine the text vector similarity between the text vector to be checked for duplicate content and each of the comparison text vectors, and determine the duplicate checking result of the text to be checked for duplicate content based on the text vector similarity.

2. The method according to claim 1, characterized in that The method further comprises: Dividing the training text into a plurality of first sentence character strings according to a preset text division rule, and dividing the document text into a plurality of second sentence character strings according to the preset text division rule; Constructing first text matrices corresponding to each of the first sentence character strings, and decomposing each of the first text matrices into a first dictionary matrix and a first sparse matrix, and constructing second text matrices corresponding to each of the second sentence character strings, and decomposing each of the second text matrices into a second dictionary matrix and a second sparse matrix; For any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix through each of the second sparse matrices to obtain a feature sparse matrix; A special text matrix is ​​constructed using the first dictionary matrix corresponding to the target sparse matrix and the feature sparse matrix, and a dimensionality reduction process is performed on the special text matrix to obtain a reduced-dimensional text vector; The first text matrix corresponding to the target sparse matrix is ​​used as input data, and the dimension-reduced text vector is used as a training label to train the word vector conversion neural network to obtain a preset word vector conversion neural network.

3. The method according to claim 2, characterized in that The step of reconstructing the target sparse matrix by using each of the second sparse matrices to obtain a feature sparse matrix corresponding to the target sparse matrix includes: Determining matrix similarities between the target sparse matrix and each of the second sparse matrices; Determine the number of second sparse matrices whose matrix similarity is greater than a second preset threshold in each of the second sparse matrices, and use the number of matrices as the eigenvalue corresponding to the target sparse matrix; A feature vector is constructed based on the eigenvalues ​​corresponding to each of the first sparse matrices, and the target sparse matrix is ​​reconstructed based on the eigenvector to obtain a feature sparse matrix.

4. The method according to claim 3, characterized in that The step of determining the matrix similarity between the target sparse matrix and each of the second sparse matrices comprises: For any second sparse matrix in each of the second sparse matrices, determining a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix; Determine the first information proportion of the first column vector in the first text matrix corresponding to the target sparse matrix, and determine the second information proportion of the second column vector in the second text matrix corresponding to the second sparse matrix, wherein the first column vector is a column vector of the target sparse matrix, and the second column vector is a column vector of the second sparse matrix; Determining an information factor based on a ratio between the first information ratio and the second information ratio; determining a column vector similarity between the first column vector and the second column vector; The matrix distance is weighted based on the information factor and the column vector similarity to obtain the matrix similarity between the target sparse matrix and the second sparse matrix.

5. The method according to claim 4, characterized in that The step of determining a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix comprises: Divide the first dictionary matrix corresponding to the target sparse matrix into a plurality of first regional matrices based on a preset matrix partitioning rule, and divide the second dictionary matrix corresponding to the second sparse matrix into a plurality of second regional matrices based on the preset matrix partitioning rule; For any target area matrix in each of the first area matrices, determining a correlation coefficient between the target area matrix and each of the second area matrices; Determine a variation coefficient value corresponding to a coefficient sequence composed of correlation coefficients corresponding to each of the first regional matrices, and determine a distance weight of each of the first regional matrices based on the variation coefficient value, wherein the variation coefficient value is negatively correlated with the distance weight; The distance weight is used as the weight of the weighted Euclidean distance to calculate the matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix.

6. The method according to any one of claims 1 to 5, characterized in that The step of constructing a duplicate text matrix corresponding to the duplicate text to be checked includes: Performing word segmentation processing on the text to be checked for duplicates to obtain multiple word character strings; Marking each character in the text to be checked for duplicates to obtain a character identifier for each character, wherein the character identifiers of identical characters in the characters are identical; The number of the multiple word strings is used as the number of rows, the number of characters of the word string with the largest number of characters in the multiple word strings is used as the number of columns, and the character identifier of each character in the multiple word strings is used as an element to construct a text matrix to be checked for duplicates.

7. A text duplication checking device, characterized in that: The device comprises: A construction module is used to obtain the text to be checked for duplicates and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates; A word vector conversion module is used to input the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain the to-be-checked duplicate text vector, wherein the word vector conversion neural network is obtained by taking a first text matrix corresponding to a training text as input data and a first text vector as a label for training, the first text vector is obtained by reducing the dimension of a special text matrix, the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text; An acquisition module, used for acquiring a comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the preset word vector conversion neural network; The comparison module is used to determine the text vector similarity between the text vector to be checked for duplicate content and each of the comparison text vectors, and determine the duplicate checking result of the text to be checked for duplicate content according to the text vector similarity.

8. A text duplication checking device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the text duplication checking method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the text duplication checking method as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the text duplication checking method as claimed in any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Text similarity detection method based on deep learning

    CN113486645A

  • Document duplicate checking method and device based on semantics

    CN114564935A

  • Text matching method and device and computer readable storage medium

    CN116127940A

  • Text comparison method and device, medium and equipment

    CN116932735A

  • Multi-dimensional text duplicate checking method based on structural similarity and semantic similarity

    CN118261138A