Text duplication checking method, device, equipment, storage medium and computer program product

Abstract

The present application discloses a text duplication checking method, apparatus, equipment, storage medium and computer program product, relates to the field of text processing technology, and discloses a text duplication checking method, including: obtaining a text to be checked for duplicates, and constructing a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates; inputting the text matrix to be checked for duplicates into a preset word vector conversion neural network to obtain a text vector to be checked for duplicates; obtaining a comparison text vector of each comparison text; determining the text vector similarity between the text vector to be checked for duplicates and each comparison text vector, and determining the duplication checking result of the text to be checked for duplicates based on the text vector similarity. In the present application, the word vector conversion neural network can learn different semantic expressions of the same semantic information during the training process, thereby introducing semantic information into the text vector, realizing semantic-based text duplication checking, and improving the accuracy of text duplication checking.

Description

Technical Field

[0001] The present application relates to the field of text processing technology, and in particular to a text duplication checking method, apparatus, device, storage medium and computer program product. Background Art

[0002] Currently, traditional text duplication detection methods based on natural language processing primarily identify duplicate content by determining whether text characters are identical. However, due to the inherent complexity of language, the same text character can have multiple meanings. The same character can convey different semantics, and different characters can also have the same semantics, such as synonyms or near-synonyms. If duplicate detection is performed solely based on whether text characters are identical, text with identical semantics but different characters may be missed. It may also result in text with identical characters but different semantics being mistakenly identified as duplicates, affecting the accuracy of the text duplication detection results.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a text duplication checking method, apparatus, device, storage medium and computer program product, aiming to solve the technical problem of low accuracy of text duplication checking results.

[0005] To achieve the above objectives, this application proposes a method for checking for duplicate text, which includes:

[0006] Obtain the text to be checked for duplicates, and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates;

[0007] Inputting the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain a to-be-checked duplicate text vector, wherein the word vector conversion neural network is trained by using a first text matrix corresponding to a training text as input data and a first text vector as a label, wherein the first text vector is obtained by dimensionality reduction of a special text matrix, and the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text;

[0008] Obtaining a comparison text vector for each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the word vector conversion neural network;

[0009] Determine the text vector similarity between the text vector to be checked for duplicates and each of the comparison text vectors, and determine the duplicate checking result of the text to be checked for duplicates based on the text vector similarity.

[0010] In one embodiment, the method further comprises:

[0011] Dividing the training text into a plurality of first sentence character strings according to a preset text division rule, and dividing the document text into a plurality of second sentence character strings according to the preset text division rule;

[0012] Constructing a first text matrix corresponding to each of the first sentence character strings, and decomposing each of the first text matrices into a first dictionary matrix and a first sparse matrix, and constructing a second text matrix corresponding to each of the second sentence character strings, and decomposing each of the second text matrices into a second dictionary matrix and a second sparse matrix;

[0013] For any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix using each of the second sparse matrices to obtain a feature sparse matrix;

[0014] A special text matrix is ​​constructed using the first dictionary matrix corresponding to the target sparse matrix and the feature sparse matrix, and a dimensionality reduction process is performed on the special text matrix to obtain a reduced-dimensionality text vector;

[0015] The first text matrix corresponding to the target sparse matrix is ​​used as input data, and the dimension-reduced text vector is used as a training label to train the word vector conversion neural network to obtain a preset word vector conversion neural network.

[0016] In one embodiment, the step of reconstructing the target sparse matrix using each of the second sparse matrices to obtain a feature sparse matrix corresponding to the target sparse matrix includes:

[0017] Determining matrix similarity between the target sparse matrix and each of the second sparse matrices;

[0018] Determine the number of second sparse matrices whose matrix similarity is greater than a second preset threshold in each of the second sparse matrices, and use the number of matrices as the eigenvalue corresponding to the target sparse matrix;

[0019] A characteristic vector is constructed based on the eigenvalues ​​corresponding to each of the first sparse matrices, and the target sparse matrix is ​​reconstructed based on the eigenvector to obtain a characteristic sparse matrix.

[0020] In one embodiment, the step of determining the matrix similarity between the target sparse matrix and each of the second sparse matrices includes:

[0021] For any second sparse matrix among the second sparse matrices, determining a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix;

[0022] Determine a first information ratio of a first column vector in a first text matrix corresponding to the target sparse matrix, and determine a second information ratio of a second column vector in a second text matrix corresponding to the second sparse matrix, wherein the first column vector is a column vector of the target sparse matrix, and the second column vector is a column vector of the second sparse matrix;

[0023] determining an information factor based on a ratio between the first information proportion and the second information proportion;

[0024] determining a column vector similarity between the first column vector and the second column vector;

[0025] The matrix distance is weighted based on the information factor and the column vector similarity to obtain the matrix similarity between the target sparse matrix and the second sparse matrix.

[0026] In one embodiment, the step of determining a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix includes:

[0027] Divide the first dictionary matrix corresponding to the target sparse matrix into a plurality of first region matrices based on a preset matrix partitioning rule, and divide the second dictionary matrix corresponding to the second sparse matrix into a plurality of second region matrices based on the preset matrix partitioning rule;

[0028] For any target region matrix in each of the first region matrices, determining a correlation coefficient between the target region matrix and each of the second region matrices;

[0029] Determining a coefficient of variation value corresponding to a coefficient sequence composed of correlation coefficients corresponding to each of the first regional matrices, and determining a distance weight of each of the first regional matrices based on the coefficient of variation value, wherein the coefficient of variation value is negatively correlated with the distance weight;

[0030] The distance weight is used as the weight of the weighted Euclidean distance to calculate the matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix.

[0031] In one embodiment, the step of constructing a duplicate text matrix corresponding to the duplicate text to be checked includes:

[0032] Performing word segmentation processing on the text to be checked for duplicates to obtain multiple word character strings;

[0033] Marking each character in the text to be checked for duplicates to obtain a character identifier for each character, wherein the character identifiers of identical characters in the characters are identical;

[0034] The matrix of texts to be checked for duplicates is constructed by taking the number of the multiple word strings as the number of rows, the number of characters of the word string with the largest number of characters in the multiple word strings as the number of columns, and the character identifier of each character in the multiple word strings as an element.

[0035] In addition, to achieve the above-mentioned purpose, the present application also proposes a text duplication checking device, which includes:

[0036] A construction module is used to obtain the text to be checked for duplicates and construct a duplicate text matrix corresponding to the text to be checked for duplicates;

[0037] A word vector conversion module is used to input the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain a to-be-checked duplicate text vector, wherein the word vector conversion neural network is obtained by taking a first text matrix corresponding to a training text as input data and a first text vector as a label for training, wherein the first text vector is obtained by dimensionality reduction of a special text matrix, and the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text;

[0038] An acquisition module, configured to acquire a comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the preset word vector conversion neural network;

[0039] The comparison module is used to determine the text vector similarity between the text vector to be checked for duplicates and each of the comparison text vectors, and determine the duplicate checking result of the text to be checked for duplicates according to the text vector similarity.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a text duplication checking device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the text duplication checking method as described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the text duplication checking method described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the text duplication checking method as described above.

[0043] In the present application, a text to be checked for duplicates is obtained, and a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates is constructed; the text matrix to be checked for duplicates is input into a preset word vector conversion neural network to obtain a text vector to be checked for duplicates, wherein the word vector conversion neural network is obtained by taking a first text matrix corresponding to a training text as input data and training with a first text vector as a label, the first text vector is obtained by dimensionality reduction of a special text matrix, the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text; a comparison text vector of each comparison text is obtained, wherein the comparison text vector is obtained by inputting the comparison text into a preset word vector conversion neural network; the text vector similarity between the text vector to be checked for duplicates and each comparison text vector is determined, and the duplicate checking result of the text to be checked for duplicates is determined based on the text vector similarity.

[0044] In this application, the first text matrix of the training text is reconstructed by using a document text with the same semantic information as the training text. The content of the document text is introduced into the first text matrix, enriching the semantic expression of the first text matrix, so that the word vector conversion neural network can learn different semantic expressions of the same semantic information during the training process, and the word vector conversion neural network can learn people's understanding of the semantic information of the text, so that it can more accurately capture the semantic similarity between texts in subsequent word vector conversions.

[0045] Therefore, when the text matrix to be checked for duplicates is input into the word vector conversion neural network for conversion, the word vector conversion neural network can generate a vector representation that can reflect the semantic information of the text. Compared with comparing text characters, this application introduces semantic information into the text vector to be checked for duplicates, which can realize semantic-based text duplication checking and improve the accuracy of text duplication checking. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0048] Figure 1 A flowchart of the first embodiment of the method for checking duplicate text in this application is provided;

[0049] Figure 2 A flowchart of the second embodiment of the method for checking duplicate text in this application is provided;

[0050] Figure 3 A schematic diagram of a simplified process of a text duplication checking method provided in one embodiment of the present application;

[0051] Figure 4 This is a schematic diagram of the module structure of the text duplication checking device according to an embodiment of the present application;

[0052] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the text duplication checking method in the embodiment of the present application.

[0053] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0055] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0056] The main solution of the embodiment of the present application is: obtain the text to be checked for duplicates, and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates; input the text matrix to be checked for duplicates into a preset word vector conversion neural network to obtain a text vector to be checked for duplicates; obtain the comparison text vectors of each comparison text, wherein the comparison text vectors are obtained by inputting the comparison text into the word vector conversion neural network; determine the text vector similarity between the text vector to be checked for duplicates and each of the comparison text vectors, and determine the duplicate checking result of the text to be checked for duplicates based on the text vector similarity.

[0057] In this embodiment, for ease of description, the following description is made with the text duplication checking device as the execution subject.

[0058] Intelligent text duplication detection, as a method for text similarity detection, has been widely used. The purpose of intelligent text duplication detection is to find texts or articles from two or more different sources and compare their similarities. This comparison is typically performed by calculating the distance and similarity between the texts to determine duplication between the texts. Traditional methods of duplication detection rely on whether the texts are identical. However, due to the presence of synonyms and antonyms, meaning that the texts represent different semantic information but represent different texts, a large amount of duplicate information will not be detected if the method is based solely on text identity. Furthermore, when texts contain many identical characters but represent completely different semantic information, incorrect duplication detection results can also be obtained. Traditional methods of text duplication detection based on natural language processing directly convert text information into text vectors based on the number of identical characters. The similarity between the two text vectors is then calculated to obtain the duplication detection results. However, since the semantic information expressed by the text data is not taken into account, the duplication detection results are unsatisfactory and have low accuracy.

[0059] The embodiment of the present application reconstructs the first text matrix of the training text by using the document text with the same semantic information as the training text, introduces the content of the document text into the first text matrix, enriches the semantic expression of the first text matrix, and enables the word vector conversion neural network to learn different semantic expressions of the same semantic information during the training process, so that the word vector conversion neural network can learn people's understanding of the semantic information of the text, so that it can more accurately capture the semantic similarity between texts in subsequent word vector conversion. Thus, when the matrix of the text to be checked for duplicates is input into the word vector conversion neural network for conversion, the word vector conversion neural network can generate a vector representation that can reflect the semantic information of the text. Compared with comparing text characters, this application introduces semantic information into the text vector to be checked for duplicates, which can realize semantic-based text duplication checking and improve the accuracy of text duplication checking.

[0060] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or text duplicate detection device capable of implementing the above functions. The following uses a text duplicate detection device as an example to illustrate this embodiment and the following embodiments.

[0061] Based on this, the present application embodiment provides a text duplication checking method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the method for checking duplicate text in this application.

[0062] In this embodiment, the text duplication checking method includes steps S10 to S40.

[0063] Step S10: obtaining the text to be checked for duplicates and constructing a text matrix corresponding to the text to be checked for duplicates.

[0064] It should be noted that the text to be checked for duplicate content is referred to as the "duplicate text to be checked," and the two-dimensional matrix corresponding to the "duplicate text to be checked" is referred to as the "duplicate text matrix." The duplicate text to be checked is converted into a numerical text matrix for subsequent machine learning and data analysis. The method for constructing the duplicate text matrix is ​​not limited and can be configured based on actual needs.

[0065] For example, in one feasible implementation, the process of constructing a matrix of texts to be checked for duplicates may be: preprocessing the texts to be checked for duplicates to obtain a plurality of standard words, which may specifically include preprocessing operations such as word segmentation, stop word removal, stem extraction, and morphological restoration; inputting each standard word into a bag-of-words model to obtain a text vector for each standard word; arranging each text vector according to the order of each standard word in the texts to be checked for duplicates to obtain a text matrix.

[0066] For example, in another feasible implementation, the process of constructing the matrix of texts to be checked for duplicates may be:

[0067] Step S101: perform word segmentation on the text to be checked for duplicates to obtain multiple word character strings.

[0068] In this embodiment, there is no restriction on the word segmentation processing method, which can be set according to actual needs. In one feasible implementation, a dictionary-based word segmentation method can be adopted, such as word segmentation based on the forward maximum matching method (Forward Maximum Matching, FMM) or word segmentation based on the backward maximum matching method (Backward Maximum Matching, BMM); in another feasible implementation, word segmentation tools can also be used for word segmentation, such as Jieba word segmentation, LAC (Lexical Analysis of Chinese, a lexical analysis tool launched by Baidu), THULAC (Tsinghua University Lexical Analyzer for Chinese, a Chinese lexical analysis tool developed and launched by the Natural Language Processing and Social Humanities Computing Laboratory of Tsinghua University), LTP (Language Technology Platform, a Chinese natural language processing platform developed by the Social Computing and Information Retrieval Research Center of Harbin Institute of Technology), etc.

[0069] Step S102 , marking each character in the text to be checked for duplicates, and obtaining a character identifier for each character, wherein the character identifiers of identical characters in the text are identical.

[0070] It's important to note that a character identifier is a unique identifier used to represent the identity of a character. Identical characters have the same character identifier. This identifier is a numerical value and provides the foundation for the subsequent construction of a numerical text matrix. The character identifier provides a unique identifier for each character, which helps accurately track and reference each character during the subsequent text matrix construction process. This allows each element (i.e., character) in the text matrix to be identified and distinguished, ensuring that no confusion or errors occur during text matrix construction, thereby improving the accuracy of text processing and analysis.

[0071] Step S103 , constructing a matrix of texts to be checked for duplicates, using the number of word strings as the number of rows, the number of characters in the word string with the largest number of characters as the number of columns, and the character identifier of each character in the word strings as an element.

[0072] It should be noted that since the number of characters in each word string may be inconsistent, there may be a situation where the number of elements in the row data is less than the number of columns. You can add 0 after the row data to ensure that all word strings have the same length, so that the data format is consistent in subsequent data processing and analysis, which facilitates subsequent calculations and processing.

[0073] Step S20: input the text matrix to be checked for duplicates into a preset word vector conversion neural network to obtain the text vector to be checked for duplicates, wherein the word vector conversion neural network is obtained by taking the first text matrix corresponding to the training text as input data and the first text vector as a label for training, the first text vector is obtained by dimensionality reduction of the special text matrix, the special text matrix is ​​obtained by reconstructing the first text matrix based on the document text, and the document text has the same semantic information as the training text.

[0074] It should be noted that the training text is the text used for model training, and the text matrix of the training text is called the first text matrix. The text with the same semantic information as the training text is called the document text, and the document text can be manually selected. Before using the training text to train the model, the first text matrix is ​​reconstructed by the document text, and the content of the document text is introduced into the first text matrix to obtain a special text matrix. The special text matrix includes different semantic expressions of the same semantics, and the special text matrix is ​​reduced in dimension to obtain the first text vector, which is used for model training. The first text matrix is ​​used as input data, and the first text vector is used as label training to obtain a word vector conversion neural network. The structure of the word vector conversion neural network is not limited here, and can be set according to actual needs. For example, it can be a neural network based on Word2Vec (Word to Vector, word vector model), a neural network based on RNN (Recurrent Neural Network, cyclic neural network) or LSTM (Long Short-Term Memory, long short-term memory network), a neural network based on DNN (Deep Neural Networks, deep neural network), etc. The matrix of text to be checked for duplicates is input into the trained word vector conversion neural network. The obtained vector is called the text to be checked for duplicates vector, which is used for subsequent text duplication checking.

[0075] Step S30: Obtain the comparison text vector of each comparison text, wherein the comparison text vector is obtained by converting the comparison text input word vector into a neural network.

[0076] It should be noted that the text to be compared with the text to be checked for duplicates is called the comparison text, and the vector of the comparison text is called the comparison text vector.

[0077] The comparison text vector is obtained by converting the comparison text input word vector into a neural network. In a feasible implementation, before the text is checked for duplicates, the comparison text input word vector is converted into a neural network to obtain the comparison text vector, the comparison text vector is stored in a database, and the comparison text vector is obtained from the database during the text check for duplicates. In the scenario where the comparison text is a fixed text, this implementation can reduce the text conversion operation during the text check for duplicates and improve the efficiency of the text check for duplicates. For example, in the scenario of comparing and checking for duplicates in papers, the comparison texts of different papers are all existing documents in the paper library and the document library. Converting the existing documents in the paper library and the document library into comparison text vectors in advance can improve the efficiency of the text check for duplicates and reduce the consumption of computing resources. In another feasible implementation, the comparison text input word vector is also converted into a neural network to obtain the comparison text vector during the text check for duplicates. It can be specifically set according to actual needs and is not limited here.

[0078] Step S40 , determining the text vector similarity between the text vector to be checked for duplicate content and each comparison text vector, and determining the duplicate checking result of the text to be checked for duplicate content based on the text vector similarity.

[0079] It should be noted that the method of determining the text vector similarity between the text vector to be checked for duplicates and each comparison text vector is not limited here and can be set according to actual needs. For example, the cosine similarity, Jaccard similarity, Euclidean distance or Manhattan distance between the text vector to be checked for duplicates and each comparison text vector can be used as the text vector similarity.

[0080] The duplicate checking result of the text to be checked is determined based on the text vector similarity, wherein the text vector similarity represents the similarity between the text to be checked for duplicates and the comparison text, that is, the higher the vector similarity, the higher the similarity between the text to be checked for duplicates and the comparison text.

[0081] In this embodiment, the first text matrix of the training text is reconstructed by using a document text with the same semantic information as the training text. The content of the document text is introduced into the first text matrix, enriching the semantic expression of the first text matrix, so that the word vector conversion neural network can learn different semantic expressions of the same semantic information during the training process, and the word vector conversion neural network can learn people's understanding of the semantic information of the text, so that it can more accurately capture the semantic similarity between texts in subsequent word vector conversions.

[0082] Therefore, when the text matrix to be checked for duplicates is input into the word vector conversion neural network for conversion, the word vector conversion neural network can generate a vector representation that can reflect the semantic information of the text. Compared with comparing text characters, this embodiment introduces semantic information into the text vector to be checked for duplicates, which can realize semantic-based text duplication checking and improve the accuracy of text duplication checking.

[0083] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 2 The text duplication checking method also includes steps S50 to S90.

[0084] Step S50 : dividing the training text into a plurality of first sentence character strings according to a preset text division rule, and dividing the document text into a plurality of second sentence character strings according to a preset text division rule.

[0085] It should be noted that the preset text segmentation rules can be set according to actual needs and are not limited here. For example, in a feasible implementation, the text can be segmented based on punctuation marks as nodes, and the text characters between two adjacent punctuation marks can be divided into a sentence string. For example, the text characters between two adjacent sentence-end punctuation marks (such as a period, exclamation mark, greeting, ellipsis, etc., which indicate the end of a sentence) can be divided into a sentence string. Compared with the sentence string segmentation based on the pauses and structural relationships within the sentence (such as semicolons, commas, etc.) as nodes, this implementation can make the sentence strings obtained by segmentation have complete semantic information, so that the word vector neural network can learn the complete semantic information of the context during the training process, enrich the semantic expression of the word vector, and improve the accuracy of the subsequent use of the word vector for text duplication checking.

[0086] Step S60: construct a first text matrix corresponding to each first sentence character string, and decompose each first text matrix into a first dictionary matrix and a first sparse matrix, and construct a second text matrix corresponding to each second sentence character string, and decompose each second text matrix into a second dictionary matrix and a second sparse matrix.

[0087] It should be noted that the method of constructing the first text matrix is ​​not limited here, and the details can be referred to step S10.

[0088] For each first text matrix, the first text matrix is ​​decomposed into a first dictionary matrix and a first sparse matrix to reduce the dimension of the first text data while retaining important information.

[0089] The first dictionary matrix represents a set of basis vectors in the first text matrix. The basis vectors can capture the common text information in the first text matrix. The first sparse matrix represents the unique text information in the first text matrix. Since different sentence strings are less likely to contain the same characters, the similarity between the dictionary matrices corresponding to different sentence strings is low. When the semantic information corresponding to different sentence strings is similar, there is similarity between the corresponding sparse matrices. Therefore, this embodiment obtains a reconstructed text matrix based on the changes in the dictionary matrix and sparse matrix corresponding to the text matrix, and further obtains the semantic information contained in the text matrix. In a specific embodiment, the method for decomposing the first text matrix is ​​not limited. For example, it can be an SVD (Singular Value Decomposition) algorithm, a K-SVD (K-Singular Value Decomposition) algorithm, etc. Similarly, the same operations as the first sentence string are performed on each second sentence string, which will not be repeated here.

[0090] Step S70 : for any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix by using each of the second sparse matrices to obtain a feature sparse matrix.

[0091] It should be noted that the target sparse matrix is ​​any one of the first sparse matrices. Since the sparse matrix represents the semantic information of the text matrix, reconstructing the target sparse matrix using the second sparse matrix can enrich the expression form of the target sparse matrix for the same semantics.

[0092] The manner of reconstructing the target sparse matrix using the second sparse matrices is not limited here.

[0093] For example, in a feasible implementation, the process of reconstructing the target sparse matrix through each second sparse matrix can be: for each second sparse matrix, based on a sparse coding algorithm (such as Lasso (Least Absolute Shrinkage and Selection Operator), OMP (Orthogonal Matching Pursuit), etc.), the column vectors in the second sparse matrix are processed to obtain the sparse coding coefficients of each column vector; and the column vectors in the second sparse matrix and the respective sparse coding coefficients are used to reconstruct the target sparse matrix in a linear combination manner.

[0094] For example, in another feasible implementation, the process of reconstructing the target sparse matrix using each second sparse matrix may include steps S701 to S703.

[0095] Step S701: Determine the matrix similarity between the target sparse matrix and each second sparse matrix.

[0096] It should be noted that matrix similarity is used to characterize the similarity between the first text matrix corresponding to the target sparse matrix and each second text matrix at the character level. Matrix similarity can reveal the degree of proximity between texts in terms of character sequence, spelling, word composition, etc. For example, if two text matrices contain a large number of identical character sequences or words, their matrix similarity may be very high. The method of determining matrix similarity is not limited here. For example, the length of the LCS (Length of Longest Common Subsequence) between the target sparse matrix and the second coefficient matrix, Jaccard similarity, or similarity based on edit distance can be used as matrix similarity.

[0097] In a feasible implementation, the process of determining matrix similarity may include A1 to A5.

[0098] A1. For any second sparse matrix among the second sparse matrices, determine a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix.

[0099] It should be noted that the following same processing is performed on any second sparse matrix among the second sparse matrices, and the following description is given by taking any second sparse matrix as an example.

[0100] The matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix represents the degree of difference between the two dictionary matrices in structure or content. A smaller matrix distance means that the two dictionary matrices are more similar, that is, the text data they represent are closer in structure or content; while a larger matrix distance indicates that the two dictionary matrices are more different, and the text data they represent are quite different in structure or content.

[0101] The method of determining the matrix distance is not limited here. For example, the Euclidean distance and the Frobenius norm (Frobenius norm) between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix can be calculated as the matrix distance.

[0102] Specifically, in a feasible implementation manner, the process of determining the matrix distance may include A11 to A13.

[0103] A11: divide the first dictionary matrix corresponding to the target sparse matrix into multiple first region matrices based on a preset matrix division rule, and divide the second dictionary matrix corresponding to the second sparse matrix into multiple second region matrices based on a preset matrix division rule.

[0104] It should be noted that the preset matrix division rule can be set according to actual needs and is not limited here.

[0105] A12: For any target region matrix in each first region matrix, determine a correlation coefficient between the target region matrix and each second region matrix.

[0106] It should be noted that the correlation coefficient represents the linear correlation strength between the target region matrix and each second region matrix, and may specifically be a Pearson correlation coefficient.

[0107] A13, determining a coefficient of variation value corresponding to a coefficient sequence composed of correlation coefficients corresponding to each first regional matrix, and determining a distance weight of each first regional matrix based on the coefficient of variation value, wherein the coefficient of variation value is negatively correlated with the distance weight.

[0108] It should be noted that the smaller the coefficient of variation value, the more uniform the target region matrix is ​​compared with the second region matrix, the more similarity information the target region matrix and the second region matrix contain, and the weights corresponding to the matrix elements of the target region matrix should be set larger for subsequent calculation of the weighted Euclidean distance.

[0109] A14 , using the distance weight as the weight of the weighted Euclidean distance, and calculating the matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix.

[0110] Specifically, the calculation formula of the weighted Euclidean distance can be:

[0111]

[0112] Where D(X, Y) is the weighted Euclidean distance between the first dictionary matrix and the second dictionary matrix, X represents the first dictionary matrix, Y represents the second dictionary matrix, X i Represents the i-th first region matrix in each first region matrix obtained by dividing the first dictionary matrix, Y i represents the i-th second region matrix among the second region matrices obtained by partitioning the second dictionary matrix, ω i Represents the weight of the i-th first region matrix.

[0113] A2, determine the proportion of first information of the first column vector in the first text matrix corresponding to the target sparse matrix, and determine the proportion of second information of the second column vector in the second text matrix corresponding to the second coefficient matrix, wherein the first column vector is the column vector of the target sparse matrix, and the second column vector is the column vector of the second sparse matrix.

[0114] It should be noted that the information ratio represents the relative size of the amount of information or importance carried by the column vector in the text matrix, the first information ratio represents the relative size of the amount of information or importance carried by the first column vector in the first text matrix corresponding to the target sparse matrix, and the second information ratio represents the relative size of the information or importance carried by the second column vector in the second text matrix corresponding to the second coefficient matrix.

[0115] The specific method for obtaining the first information ratio of the first column vector is: setting the first column vector to 0, and using the target sparse matrix after the first column vector is set to 0 as the new sparse matrix; reconstructing the new sparse matrix and the first dictionary matrix of the target coefficient matrix to obtain a new text matrix; calculating the inverse function of the Euclidean distance between the new text matrix and the first text matrix corresponding to the target coefficient matrix as the information ratio of the first column vector.

[0116] The information ratio of the second column vector can be calculated by referring to the calculation process of the first column vector, which will not be repeated here.

[0117] A3, determining an information factor based on a ratio between the first information ratio and the second information ratio.

[0118] It should be noted that the information factor represents the relative importance of the first and second information proportions and is used to weight the matrix distances in subsequent processes to obtain matrix similarity. The information factor can be the ratio of the first information proportion divided by the second information proportion, or the ratio of the second information proportion divided by the first information proportion, without limitation.

[0119] A4, determining the column vector similarity between the first column vector and the second column vector.

[0120] It should be noted that the method of calculating column vector similarity is not limited here. For example, the cosine similarity, Jaccard similarity, Euclidean distance or Manhattan distance between the first column vector and the second column vector can be used as the column vector similarity.

[0121] A5, performing weighted processing on the matrix distance based on the information factor and the column vector similarity, and obtaining the matrix similarity between the target sparse matrix and the second sparse matrix.

[0122] Step S702 : determining the number of second sparse matrices whose matrix similarity in each second sparse matrix is ​​greater than a second preset threshold, and using the number of matrices as the eigenvalue corresponding to the target sparse matrix.

[0123] It should be noted that by setting a similarity threshold to extract eigenvalues, it is possible to extract information related to the target sparse matrix from a large number of second sparse matrices and simplify this information into a single eigenvalue, which can reduce the operational complexity in the subsequent reconstruction of the sparse matrix and improve the efficiency of the reconstructed matrix. It can also filter out sparse matrices with low similarity to the target sparse matrix, thereby reducing the impact of noise and outliers on the subsequent reconstruction of the sparse matrix, so as to improve the overall quality and accuracy of the subsequent reconstruction of the sparse matrix.

[0124] Step S703 : constructing eigenvectors based on the eigenvalues ​​corresponding to the first sparse matrices, and reconstructing the target sparse matrix based on the eigenvectors to obtain a characteristic sparse matrix.

[0125] It should be noted that the eigenvector and the eigensparse matrix provide richer information than the original sparse matrix, and can better describe the similarities and differences between texts, so as to improve the accuracy of subsequent text duplication checking.

[0126] Step S80: construct a special text matrix using the first dictionary matrix and the feature sparse matrix corresponding to the target sparse matrix, and perform dimensionality reduction processing on the special text matrix to obtain a reduced-dimensional text vector.

[0127] Using the feature sparse matrix and the first dictionary matrix for inverse decomposition, the special text matrix is ​​reconstructed. Matrix dimensionality reduction is then performed on the special text matrix to obtain a reduced-dimensional text vector. This converts high-dimensional data into a low-dimensional space, significantly reducing the data dimension. This can reduce computing resource overhead and improve the efficiency of subsequent text duplication checks. Furthermore, during the dimensionality reduction process, some noise and redundant features are typically removed, making the data purer and easier to analyze, thereby improving the accuracy and reliability of subsequent text duplication checks.

[0128] It should be noted that there is no restriction on the method of performing matrix dimensionality reduction on special text matrices. For example, it can be based on principal component analysis (PCA), singular value decomposition (SVD) or truncated singular value decomposition (Truncated SVD), non-negative matrix factorization (NMF), etc., and can be set specifically according to actual needs.

[0129] Step S90: Using the first text matrix corresponding to the target sparse matrix as input data and the dimension-reduced text vector as a training label, the word vector conversion neural network to be trained is trained to obtain a preset word vector conversion neural network.

[0130] In this embodiment, the training text is divided into multiple first sentence character strings according to a preset text division rule, and the document text is divided into multiple second sentence character strings according to a preset text division rule; a first text matrix corresponding to each first sentence character string is constructed, and each first text matrix is ​​decomposed into a first dictionary matrix and a first sparse matrix, and a second text matrix corresponding to each second sentence character string is constructed, and each second text matrix is ​​decomposed into a second dictionary matrix and a second sparse matrix; for any target sparse matrix in each first sparse matrix, the target sparse matrix is ​​reconstructed by each second sparse matrix to obtain a feature sparse matrix; a special text matrix is ​​constructed using the first dictionary matrix and the feature sparse matrix corresponding to the target sparse matrix, and the special text matrix is ​​subjected to dimensionality reduction processing to obtain a reduced dimensionality text vector; the first text matrix corresponding to the target sparse matrix is ​​used as input data, and the reduced dimensionality text vector is used as a training label to train the word vector conversion neural network to obtain a preset word vector conversion neural network.

[0131] In this implementation, the first text matrix of the training text is reconstructed by using a document text with the same semantic information as the training text. The content of the document text is introduced into the first text matrix, enriching the semantic expression of the first text matrix, so that the word vector conversion neural network can learn different semantic expressions of the same semantic information during the training process, and the word vector conversion neural network can learn people's understanding of the semantic information of the text, so that it can more accurately capture the semantic similarity between texts in subsequent word vector conversions, so as to improve the accuracy of subsequent text duplication checking based on word vectors.

[0132] For example, in order to help understand the technical concept or technical principle of the text duplication checking method after combining this embodiment with the above-mentioned embodiment 1 and embodiment 2, please refer to Figure 3 , Figure 3 A brief flowchart of a text duplication checking method is provided, as follows:

[0133] Step S001: Obtain sample text and document text. The document text and the training text have the same semantic information. Specifically, the sample text and the document text can be obtained through manual classification by professionals. Texts with the same semantic information are texts that express the same meaning. Professionals can analyze and judge based on existing synonyms or synonyms and sentence structure transformations.

[0134] Step S002: Divide the training text into a plurality of first sentence strings according to a preset text division rule, and construct a first text matrix corresponding to each first sentence string. Divide the document text into a plurality of second sentence strings according to a preset text division rule, and construct a second text matrix corresponding to each second sentence string.

[0135] In step (2.1), the character string between two adjacent periods in the text data is recorded as a sentence character string.

[0136] Step (2.2): Perform Jieba word segmentation on any sentence string to obtain several words in the sentence string, denoted as the word string, so as to construct a two-dimensional matrix, that is, the text matrix. The specific process of constructing the two-dimensional matrix can be: obtain the maximum value of the number of characters contained in all word strings in the sentence string, denoted as the first value; denote the number of word strings included in the sentence string as the second value; use the first value as the number of columns of the two-dimensional matrix and the second value as the number of rows of the two-dimensional matrix; regard the same characters in the sentence string as one category, obtain the number of types of characters in the sentence string, start marking each type of character from the number 1, denoted as the marking number (that is, the character identifier); denote the sequence formed by the marking numbers of the characters included in the word string as the row data of the word string; according to the order of the word strings in the sentence string, use the row data of the word strings as the row vectors of the two-dimensional matrix; when the number of columns of the row data is different from the number of columns of the two-dimensional matrix, perform a padding operation with 0 at the end of the row data; denote the two-dimensional matrix formed by the row data of all word strings as the text matrix of the sentence string; denote the text matrix of the sentence string in the first text as the first text matrix, and denote the text matrix of the sentence string in the second text as the second text matrix. For example, if there is a sentence string "Construct a text matrix", the included word strings are "Construct" and "text matrix" respectively. Denote the character "构" as 1, the character "建" as 2, the character "文" as 3, the character "本" as 4, the character "矩" as 5, and the character "阵" as 6. Then the row data corresponding to the word string "Construct" is [1, 2], and the row data corresponding to the word string "text matrix" is [3, 4, 5, 6].

[0137] Step S003: Reconstruct the first text matrix to obtain a special text matrix. Specifically, for any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix through each of the second sparse matrices to obtain a feature sparse matrix, and construct a special text matrix using the first dictionary matrix corresponding to the target sparse matrix and the feature sparse matrix.

[0138] In step (3.1), the K-SVD decomposition algorithm is used to decompose any text matrix into a corresponding dictionary matrix and a sparse matrix. The sparse matrix corresponding to the first text matrix is ​​recorded as the first sparse matrix, the sparse matrix corresponding to the second text matrix is ​​recorded as the second sparse matrix, the column vectors in any sparse matrix are recorded as sparse column vectors, and the sparse column vectors in the first sparse matrix and the second sparse matrix are recorded as the first column vector and the second column vector, respectively, to obtain a plurality of first column vectors and second column vectors. The dictionary matrix represents the common text information in the text matrix, while the sparse matrix represents the unique text information in the text matrix. Since the probability of different sentence strings containing the same characters is low, the similarity between the dictionary matrices corresponding to different sentence strings is low. However, when the semantic information corresponding to different sentence strings is similar, there is similarity between the corresponding sparse matrices. Therefore, this embodiment obtains a reconstructed text matrix based on the changes in the dictionary matrix and the sparse matrix corresponding to the text matrix, and further obtains the semantic information contained in the text matrix. When the semantic information between different sentence strings is similar, the sparse matrices corresponding to the two sentence strings are similar. The sparse matrix is ​​reset based on the similarity between the sparse matrices, and then the text matrix is ​​reconstructed to introduce semantic information. It should be noted that by obtaining the similarity between the sparse column vectors in the first sparse matrix and the sparse column vectors in the second sparse matrix, the best matching combination of the column vectors of the sparse matrix is ​​obtained, and this matching combination reflects the best semantic information matching result.

[0139] Step (3.2) constructs a bipartite graph. The method for constructing a bipartite graph is as follows: the first column vector is used as the node on the left side of the bipartite graph, and the second column vector is used as the node on the right side of the bipartite graph, and the connection between the left node and the right node in the bipartite graph is called an edge.

[0140] Step (3.3) obtains the similarity between the i-th left node and the j-th right node in the bipartite graph (that is, the matrix similarity between the target sparse matrix and each second sparse matrix). The specific calculation method is:

[0141]

[0142] Among them, α ij represents the similarity between the i-th left node and the j-th right node in the bipartite graph; τ ij represents the difference between the i-th left node and the j-th right node; λ i Indicates the information ratio of the column vector corresponding to the i-th left node, λ j represents the information ratio of the sparse column vector corresponding to the j-th right node; cos(i, j) represents the cosine similarity between the sparse column vectors corresponding to the i-th left node and the j-th right node.

[0143] It should be noted that the weighted Euclidean distance between the dictionary matrices of the text matrices corresponding to the sparse matrices of any left and right nodes is obtained, which is recorded as the difference between the left and right nodes. When two sentence strings contain the same characters, the two text matrices are identical, and the dictionary matrices after K-SVD decomposition may also contain the same information. Therefore, if only the Euclidean distance is used as the metric, significant differences will be observed. Therefore, the structural information of the dictionary matrix should be analyzed accordingly. In this embodiment, the adaptive Euclidean distance is calculated by assigning a weight to each element in the dictionary matrix, wherein the dictionary matrix is ​​divided into 8 equal regions, wherein the Pearson correlation coefficient is calculated between a single region of the first text dictionary matrix and each region of the second text matrix, and then the Pearson correlation coefficient sequence of a single region of the first text dictionary matrix can be obtained, and the coefficient of variation of the sequence is calculated. If the coefficient of variation is smaller, it indicates that this region in the first text dictionary matrix is ​​more uniform than that in the second text dictionary matrix, and it may contain more similarity information. Then, the matrix element of the corresponding region in the first text dictionary matrix should be set larger when obtaining the weight, that is, the matrix element of the corresponding region is an inverse proportional function of the coefficient of variation, and then a weighted Euclidean distance is performed. Distance calculation (i.e., dividing the first dictionary matrix corresponding to the target sparse matrix into a plurality of first region matrices based on a preset matrix partitioning rule, and dividing the second dictionary matrix corresponding to the second sparse matrix into a plurality of second region matrices based on a preset matrix partitioning rule; for any target region matrix in each first region matrix, determining the correlation coefficient between the target region matrix and each second region matrix; determining the coefficient of variation corresponding to the coefficient sequence composed of the correlation coefficients corresponding to each first region matrix, and determining the distance weight of each first region matrix based on the coefficient of variation, wherein the coefficient of variation is negatively correlated with the distance weight; using the distance weight as the weight of the weighted Euclidean distance, calculating the matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix). It should be noted that the smaller the difference between the dictionary matrices corresponding to the first text matrix and the second text matrix, the more characters the first text and the second text include in common. Therefore, it is more necessary to consider the difference between the sparse column vectors corresponding to the first text matrix and the second text matrix to reflect the representation ability of the two sparse column vectors for the semantic information of the text, that is, it is necessary to consider the size of the cosine similarity between the two sparse column vectors. It should be noted that the specific method for obtaining the information ratio of the column vector is as follows: set the sparse column vector of any sparse matrix to 0, record the sparse matrix after the sparse column vector is set to 0 as the new sparse matrix of the text matrix, combine the new sparse matrix of the text matrix and the dictionary matrix, and use the K-SVD decomposition algorithm to reconstruct the reconstructed text matrix as the new text matrix corresponding to the sparse column vector, and take the inverse of the Euclidean distance d between the text matrix of the sparse matrix to which the sparse column vector belongs and the new text matrix It should be noted that this embodiment uses the cosine similarity between sparse column vectors to characterize the connection characteristics between two sparse column vectors. The greater the connection between the two sparse column vectors, the more similar the semantic information represented by the two sparse column vectors is.

[0144] In step (3.4), the similarity between the left node and the right node is used as the edge weight of the edge connecting the left node and the right node in the bipartite graph, and the number of right nodes with any left node and all edge weights greater than the preset edge weight threshold is obtained, which is recorded as the eigenvalue of the first column vector corresponding to the left node (that is, the number of matrices of the second sparse matrix whose matrix similarity in each second sparse matrix is ​​greater than the second preset threshold, and the number of matrices is used as the eigenvalue corresponding to the target sparse matrix). It should be noted that the edge weight threshold can be preset to 0.65 based on experience, which can be adjusted according to specific circumstances. This embodiment does not specifically limit it. It should be noted that the size of the eigenvalue of the first column vector represents the number of second column vectors with high similarity to the second sparse matrix, reflecting the degree of similarity between the first column vector and the second sparse matrix. The larger the eigenvalue, the greater the degree of similarity between the first column vector and the second sparse matrix.

[0145] Step (3.5) obtains the eigenvalues ​​of all first column vectors in the first sparse matrix, and performs linear normalization on all eigenvalues ​​to obtain normalized eigenvalues. The column vector formed by all normalized eigenvalues ​​of the first sparse matrix is ​​recorded as the eigenvector of the first sparse matrix. The first sparse matrix is ​​multiplied by the eigenvector to obtain the characteristic sparse matrix of the first text matrix. The characteristic sparse matrix and the dictionary matrix of the first text matrix are combined and reconstructed using the K-SVD decomposition algorithm to obtain the special text matrix of the first text matrix. All special text matrices of the sample text are obtained, and each sentence string corresponds to a special text matrix (that is, a eigenvector is constructed based on the eigenvalue corresponding to each first sparse matrix, and the target sparse matrix is ​​reconstructed based on the eigenvector to obtain a characteristic sparse matrix). It should be noted that after manual classification, the special text matrix of the text matrix obtained reflects the specific quantitative features of the text matrix when describing the semantic information of the sentence string corresponding to the text matrix under the same semantic information, which facilitates the subsequent analysis of the similarity between the sentence strings in terms of semantic information.

[0146] Step S004: training the word vector conversion neural network. The specific process is: step (4.1), using the first dictionary matrix and the feature sparse matrix corresponding to the target sparse matrix to construct a special text matrix, and performing dimensionality reduction processing on the special text matrix to obtain a reduced-dimensional text vector. Specifically, the PCA algorithm can be used to reduce the dimensionality of the special text matrix to a one-dimensional vector to obtain a reduced-dimensional text vector. Step (4.2), using the first text matrix corresponding to the target sparse matrix as input data and the reduced-dimensional text vector as the training label, the word vector conversion neural network to be trained is trained to obtain a preset word vector conversion neural network. It should be noted that the word vector conversion neural network can be a single hidden layer DNN neural network, and the root mean square error function is used as the loss function of the neural network for training.

[0147] Step S005: Use the trained word vector conversion neural network to perform text duplication check. The specific process is: Step (5.1), obtain the text to be checked for duplicates, and construct the text matrix to be checked for duplicates corresponding to the text to be checked for duplicates. Step (5.2), input the text matrix to be checked for duplicates into the trained word vector conversion neural network to obtain the text vector to be checked for duplicates. Step (5.3), obtain the comparison text vectors of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the word vector conversion neural network. Step (5.4), determine the text vector similarity between the text vector to be checked for duplicates and each comparison text vector, and determine the duplicate checking result of the text to be checked for duplicates based on the text vector similarity. It should be noted that when the cosine similarity between the text vector to be checked for duplicates and the comparison text vector is used as the text vector similarity, when the text vector similarity is greater than the preset cosine similarity threshold, the sentence string corresponding to the duplicate checking text vector is marked as repeated. Specifically, the size of the cosine similarity between the duplicate check vector and the document vector reflects the degree of similarity between the corresponding semantic information of the corresponding sentence strings. The larger the cosine similarity, the closer the semantic information expressed by the sentence strings is. The cosine similarity threshold can be preset to 0.78 based on experience and can be adjusted according to actual conditions. This embodiment does not make specific limitations.

[0148] In this embodiment, the vector conversion of text information is performed by training a word vector conversion neural network, and then the vectors are compared to realize text duplication detection, which avoids the shortcomings of the traditional word vector conversion network that only converts according to word frequency without considering the semantic information of the text. The word vector conversion neural network can describe the text data according to the semantic information of the text data, thereby improving the influence of the semantic information of the text data in the text duplication detection process and further improving the accuracy of the text duplication detection.

[0149] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the method of checking for duplicate text in this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0150] This application also provides a text duplication checking device, please refer to Figure 4 , the text duplication checking device comprises:

[0151] A construction module 10 is used to obtain a text to be checked for duplicates and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates;

[0152] A word vector conversion module 20 is used to input the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain a to-be-checked duplicate text vector, wherein the word vector conversion neural network is obtained by using a first text matrix corresponding to a training text as input data and a first text vector as a label for training, wherein the first text vector is obtained by dimensionality reduction of a special text matrix, and the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text;

[0153] An acquisition module 30 is configured to acquire a comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the preset word vector conversion neural network;

[0154] The comparison module 40 is configured to determine the text vector similarity between the text vector to be checked for duplicate content and each of the comparison text vectors, and determine a duplicate checking result of the text to be checked for duplicate content based on the text vector similarity.

[0155] Optionally, the text duplication checking device further includes a training module for:

[0156] Dividing the training text into a plurality of first sentence character strings according to a preset text division rule, and dividing the document text into a plurality of second sentence character strings according to the preset text division rule;

[0157] Constructing a first text matrix corresponding to each of the first sentence character strings, and decomposing each of the first text matrices into a first dictionary matrix and a first sparse matrix, and constructing a second text matrix corresponding to each of the second sentence character strings, and decomposing each of the second text matrices into a second dictionary matrix and a second sparse matrix;

[0158] For any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix using each of the second sparse matrices to obtain a feature sparse matrix;

[0159] A special text matrix is ​​constructed using the first dictionary matrix corresponding to the target sparse matrix and the feature sparse matrix, and a dimensionality reduction process is performed on the special text matrix to obtain a reduced-dimensionality text vector;

[0160] The first text matrix corresponding to the target sparse matrix is ​​used as input data, and the dimension-reduced text vector is used as a training label to train the word vector conversion neural network to obtain a preset word vector conversion neural network.

[0161] Optionally, the training module is further used to:

[0162] Determining matrix similarity between the target sparse matrix and each of the second sparse matrices;

[0163] Determine the number of second sparse matrices whose matrix similarity is greater than a second preset threshold in each of the second sparse matrices, and use the number of matrices as the eigenvalue corresponding to the target sparse matrix;

[0164] A characteristic vector is constructed based on the eigenvalues ​​corresponding to each of the first sparse matrices, and the target sparse matrix is ​​reconstructed based on the eigenvector to obtain a characteristic sparse matrix.

[0165] Optionally, the training module is further used to:

[0166] For any second sparse matrix among the second sparse matrices, determining a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix;

[0167] Determine a first information ratio of a first column vector in a first text matrix corresponding to the target sparse matrix, and determine a second information ratio of a second column vector in a second text matrix corresponding to the second sparse matrix, wherein the first column vector is a column vector of the target sparse matrix, and the second column vector is a column vector of the second sparse matrix;

[0168] determining an information factor based on a ratio between the first information proportion and the second information proportion;

[0169] determining a column vector similarity between the first column vector and the second column vector;

[0170] The matrix distance is weighted based on the information factor and the column vector similarity to obtain the matrix similarity between the target sparse matrix and the second sparse matrix.

[0171] Optionally, the training module is further used to:

[0172] Divide the first dictionary matrix corresponding to the target sparse matrix into a plurality of first region matrices based on a preset matrix partitioning rule, and divide the second dictionary matrix corresponding to the second sparse matrix into a plurality of second region matrices based on the preset matrix partitioning rule;

[0173] For any target region matrix in each of the first region matrices, determining a correlation coefficient between the target region matrix and each of the second region matrices;

[0174] Determining a coefficient of variation value corresponding to a coefficient sequence composed of correlation coefficients corresponding to each of the first regional matrices, and determining a distance weight of each of the first regional matrices based on the coefficient of variation value, wherein the coefficient of variation value is negatively correlated with the distance weight;

[0175] The distance weight is used as the weight of the weighted Euclidean distance to calculate the matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix.

[0176] Optionally, the building module 10 is further configured to:

[0177] Performing word segmentation processing on the text to be checked for duplicates to obtain multiple word character strings;

[0178] Marking each character in the text to be checked for duplicates to obtain a character identifier for each character, wherein the character identifiers of identical characters in the characters are identical;

[0179] The matrix of texts to be checked for duplicates is constructed by taking the number of the multiple word strings as the number of rows, the number of characters of the word string with the largest number of characters in the multiple word strings as the number of columns, and the character identifier of each character in the multiple word strings as an element.

[0180] The text duplication checking device provided by this application, which employs the text duplication checking method of the above-mentioned embodiment, can resolve the technical problem of low accuracy in text duplication checking results. Compared with the prior art, the beneficial effects of the text duplication checking device provided by this application are the same as those of the text duplication checking method provided by the above-mentioned embodiment, and the other technical features of the text duplication checking device are the same as those disclosed in the above-mentioned embodiment method, and are not further described here.

[0181] The present application provides a text duplication checking device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the text duplication checking method in the above-mentioned embodiment one.

[0182] Reference below Figure 5, which shows a schematic structural diagram of a text duplicate checking device suitable for implementing the embodiments of the present application. The text duplicate checking device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The text duplication checking device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0183] like Figure 5 As shown, the text duplication checking device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the text duplication checking device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. The communication device 1009 can allow the text duplicate checking device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a text duplicate checking device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0184] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0185] The text duplication checking device provided by this application, which employs the text duplication checking method of the above-mentioned embodiment, can solve the technical problem of low accuracy of text duplication checking results. Compared with the prior art, the beneficial effects of the text duplication checking device provided by this application are the same as those of the text duplication checking method provided by the above-mentioned embodiment, and the other technical features of the text duplication checking device are the same as those disclosed in the above-mentioned embodiment, and are not further described here.

[0186] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0187] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0188] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the text duplication checking method in the above-mentioned embodiment.

[0189] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0190] The above-mentioned computer-readable storage medium may be included in the text duplication checking device; or it may exist independently without being assembled into the text duplication checking device.

[0191] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the text duplication checking device, the text duplication checking device is enabled to: obtain the text to be checked for duplicates, and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates; input the text matrix to be checked for duplicates into a preset word vector conversion neural network to obtain a text vector to be checked for duplicates, wherein the word vector conversion neural network is obtained by taking a first text matrix corresponding to a training text as input data and training with a first text vector as a label, the first text vector is obtained by dimensionality reduction of a special text matrix, and the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text; obtain a comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the word vector conversion neural network; determine the text vector similarity between the text vector to be checked for duplicates and each of the comparison text vectors, and determine the duplication checking result of the text to be checked for duplicates based on the text vector similarity.

[0192] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0193] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0194] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0195] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., a computer program) for executing the above-mentioned text duplication checking method, thereby resolving the technical problem of low accuracy in text duplication checking results. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the text duplication checking method provided in the above-mentioned embodiment, and are not further elaborated here.

[0196] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned text duplication checking method when executed by a processor.

[0197] The computer program product provided by this application can solve the technical problem of low accuracy of text duplicate checking results. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the text duplicate checking method provided by the above embodiment, and will not be repeated here.

[0198] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A text duplication checking method, characterized in that: The method includes: Obtain the text to be checked for duplicates, and construct a text matrix to be checked for duplicates corresponding to the text to be checked for duplicates; Inputting the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain a to-be-checked duplicate text vector, wherein the word vector conversion neural network is trained by using a first text matrix corresponding to a training text as input data and a first text vector as a label, wherein the first text vector is obtained by dimensionality reduction of a special text matrix, and the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text; Obtaining a comparison text vector for each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the word vector conversion neural network; Determining the text vector similarity between the text vector to be checked for duplicates and each of the comparison text vectors, and determining a duplicate checking result of the text to be checked for duplicates based on the text vector similarity; The method further comprises: Dividing the training text into a plurality of first sentence character strings according to a preset text division rule, and dividing the document text into a plurality of second sentence character strings according to the preset text division rule; Constructing a first text matrix corresponding to each of the first sentence character strings, and decomposing each of the first text matrices into a first dictionary matrix and a first sparse matrix, and constructing a second text matrix corresponding to each of the second sentence character strings, and decomposing each of the second text matrices into a second dictionary matrix and a second sparse matrix; For any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix using each of the second sparse matrices to obtain a feature sparse matrix; A special text matrix is ​​constructed using the first dictionary matrix corresponding to the target sparse matrix and the feature sparse matrix, and a dimensionality reduction process is performed on the special text matrix to obtain a reduced-dimensionality text vector; The first text matrix corresponding to the target sparse matrix is ​​used as input data, and the dimension-reduced text vector is used as a training label to train the word vector conversion neural network to obtain a preset word vector conversion neural network.

2. The method according to claim 1, wherein The step of reconstructing the target sparse matrix by using each of the second sparse matrices to obtain a feature sparse matrix corresponding to the target sparse matrix includes: Determining matrix similarity between the target sparse matrix and each of the second sparse matrices; Determine the number of second sparse matrices whose matrix similarity is greater than a second preset threshold in each of the second sparse matrices, and use the number of matrices as the eigenvalue corresponding to the target sparse matrix; A characteristic vector is constructed based on the eigenvalues ​​corresponding to each of the first sparse matrices, and the target sparse matrix is ​​reconstructed based on the eigenvector to obtain a characteristic sparse matrix.

3. The method according to claim 2, wherein The step of determining the matrix similarity between the target sparse matrix and each of the second sparse matrices comprises: For any second sparse matrix among the second sparse matrices, determining a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix; Determine a first information ratio of a first column vector in a first text matrix corresponding to the target sparse matrix, and determine a second information ratio of a second column vector in a second text matrix corresponding to the second sparse matrix, wherein the first column vector is a column vector of the target sparse matrix, and the second column vector is a column vector of the second sparse matrix; determining an information factor based on a ratio between the first information proportion and the second information proportion; determining a column vector similarity between the first column vector and the second column vector; The matrix distance is weighted based on the information factor and the column vector similarity to obtain the matrix similarity between the target sparse matrix and the second sparse matrix.

4. The method according to claim 3, wherein The step of determining a matrix distance between a second dictionary matrix corresponding to the second sparse matrix and a first dictionary matrix corresponding to the target sparse matrix includes: Divide the first dictionary matrix corresponding to the target sparse matrix into a plurality of first region matrices based on a preset matrix partitioning rule, and divide the second dictionary matrix corresponding to the second sparse matrix into a plurality of second region matrices based on the preset matrix partitioning rule; For any target region matrix in each of the first region matrices, determining a correlation coefficient between the target region matrix and each of the second region matrices; Determining a coefficient of variation value corresponding to a coefficient sequence composed of correlation coefficients corresponding to each of the first regional matrices, and determining a distance weight of each of the first regional matrices based on the coefficient of variation value, wherein the coefficient of variation value is negatively correlated with the distance weight; The distance weight is used as the weight of the weighted Euclidean distance to calculate the matrix distance between the second dictionary matrix corresponding to the second sparse matrix and the first dictionary matrix corresponding to the target sparse matrix.

5. The method according to any one of claims 1 to 4, characterized in that The step of constructing a duplicate text matrix corresponding to the duplicate text to be checked includes: Performing word segmentation processing on the text to be checked for duplicates to obtain multiple word character strings; Marking each character in the text to be checked for duplicates to obtain a character identifier for each character, wherein the character identifiers of identical characters in the characters are identical; The matrix of texts to be checked for duplicates is constructed by taking the number of the multiple word strings as the number of rows, the number of characters of the word string with the largest number of characters in the multiple word strings as the number of columns, and the character identifier of each character in the multiple word strings as an element.

6. A text duplication checking device, characterized in that: The device comprises: A construction module is used to obtain the text to be checked for duplicates and construct a duplicate text matrix corresponding to the text to be checked for duplicates; A word vector conversion module is used to input the to-be-checked duplicate text matrix into a preset word vector conversion neural network to obtain a to-be-checked duplicate text vector, wherein the word vector conversion neural network is obtained by taking a first text matrix corresponding to a training text as input data and a first text vector as a label for training, wherein the first text vector is obtained by dimensionality reduction of a special text matrix, and the special text matrix is ​​obtained by reconstructing the first text matrix based on a document text, and the document text has the same semantic information as the training text; An acquisition module, configured to acquire a comparison text vector of each comparison text, wherein the comparison text vector is obtained by inputting the comparison text into the preset word vector conversion neural network; A comparison module is used to determine the text vector similarity between the text vector to be checked for duplicates and each of the comparison text vectors, and determine the duplicate checking result of the text to be checked for duplicates based on the text vector similarity; A training module is used to divide the training text into multiple first sentence character strings according to a preset text division rule, and to divide the document text into multiple second sentence character strings according to the preset text division rule; construct a first text matrix corresponding to each of the first sentence character strings, and decompose each of the first text matrices into a first dictionary matrix and a first sparse matrix, and construct a second text matrix corresponding to each of the second sentence character strings, and decompose each of the second text matrices into a second dictionary matrix and a second sparse matrix; for any target sparse matrix in each of the first sparse matrices, reconstruct the target sparse matrix through each of the second sparse matrices to obtain a feature sparse matrix; use the first dictionary matrix corresponding to the target sparse matrix and the feature sparse matrix to construct a special text matrix, perform dimensionality reduction processing on the special text matrix to obtain a reduced dimensionality text vector; use the first text matrix corresponding to the target sparse matrix as input data, and the reduced dimensionality text vector as training label to train the word vector conversion neural network to obtain a preset word vector conversion neural network.

7. A text duplication checking device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the text duplication checking method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the text duplication checking method according to any one of claims 1 to 5 are implemented.

9. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the text duplication checking method according to any one of claims 1 to 5 are implemented.

Images