Text similarity determination method and apparatus, and computer-readable storage medium
By combining bag-of-words features, semantic features, and contrastive statistical features, and utilizing logistic regression and classification models, the problem of inaccurate text similarity calculation in existing methods is solved, achieving more efficient and accurate text similarity determination.
Patent Information
- Application Number
- CN202010603116.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-06-29
- Publication Date
- 2026-01-27
- Estimated Expiration
- 2040-06-29
AI Technical Summary
Existing text similarity calculation methods mainly rely on single feature extraction, resulting in inaccurate text similarity determination.
By combining bag-of-words features, semantic features, and contrastive statistical features, and utilizing pre-trained logistic regression and classification models, text similarity is comprehensively determined.
It improves the accuracy and robustness of text similarity determination and reduces computational costs.
Smart Images
Figure CN113935387B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus and computer-readable storage medium for determining text similarity. Background Technology
[0002] Text similarity calculation is a very fundamental and crucial problem in the field of natural language processing, and it can be used in scenarios such as information retrieval, machine translation, automatic question answering systems, and text mining.
[0003] Current methods for calculating text similarity primarily rely on extracting text features and then comparing these features to determine text similarity. Examples include extracting bag-of-words features for comparison and extracting TF-IDF (Term Frequency–Inverse Document Frequency) features for comparison. Summary of the Invention
[0004] The inventors discovered that existing methods for calculating text similarity primarily extract and compare single features of the text, such as calculating the similarity between two feature vectors or comparing features using models. A single feature cannot comprehensively reflect the characteristics of the text, easily leading to inaccurate determination of text similarity.
[0005] One of the technical problems this disclosure aims to solve is to propose a new method for determining text similarity and improve the accuracy of text similarity determination.
[0006] According to some embodiments of this disclosure, a method for determining text similarity is provided, comprising: determining the similarity between a first text and a second text based on a first bag-of-words feature of a first text and a second bag-of-words feature of a second text, as a preliminary similarity; determining a first semantic feature of the first text based on the importance of each word in the first text, and determining a second semantic feature of the second text based on the importance of each word in the second text; inputting the preliminary similarity, the first semantic feature, the second semantic feature, and the comparative statistical features of the first text and the second text into a pre-trained classification model to obtain the similarity between the first text and the second text; wherein the comparative statistical features include at least one of the following: the distance between the first text and the second text, the word comparison results of the first text and the second text, and the character comparison results of the first text and the second text.
[0007] In some embodiments, determining the similarity between the first text and the second text based on the first bag-of-words features of the first text and the second bag-of-words features of the second text includes: performing a preset first operation on the first bag-of-words features and the second bag-of-words features to obtain a first feature matrix, wherein the first bag-of-words features and the second bag-of-words features are matrices; performing a preset second operation on the first bag-of-words features of the first text and the second bag-of-words features of the second text to obtain a second feature matrix; inputting the first feature matrix into a pre-trained first logistic regression model to obtain a first similarity between the first text and the second text; and inputting the second feature matrix into a pre-trained second logistic regression model to obtain a second similarity between the first text and the second text.
[0008] In some embodiments, the first operation is preset to be the subtraction of the first bag-of-words features and the second bag-of-words features; the second operation is preset to be the dot product of the first bag-of-words features and the second bag-of-words features.
[0009] In some embodiments, determining the first semantic feature of the first text based on the importance of each word in the first text includes: determining the importance of each word in the first text and generating a first importance matrix; performing singular value decomposition on the first importance matrix to obtain a dimension-reduced first importance matrix, which is used as the first semantic feature; or, determining the second semantic feature of the second text based on the importance of each word in the second text includes: determining the importance of each word in the second text and generating a second importance matrix; performing singular value decomposition on the second importance matrix to obtain a dimension-reduced second importance matrix, which is used as the second semantic feature.
[0010] In some embodiments, the importance of each word in the first text is determined based on the frequency of each word appearing in the first text and the frequency of each word appearing in the corpus; or, the importance of each word in the second text is determined based on the frequency of each word appearing in the second text and the frequency of each word appearing in the corpus.
[0011] In some embodiments, the method further includes: segmenting the first text and the second text into words respectively; determining the word vectors of each word in the first text and the word vectors of each word in the second text respectively; determining the text vector of the first text based on the word vectors of each word in the first text, and determining the text vector of the second text based on the word vectors of each word in the second text; determining the spatial distance between the text vectors of the first text and the text vectors of the second text as the distance between the first text and the second text; wherein the spatial distance includes at least one of Manhattan distance, Jaccard distance, Canberra distance, Euclidean distance, Minkowski distance, and Brectis distance.
[0012] In some embodiments, the method further includes: determining the similarity between the strings of the first text and the strings of the second text based on the edit distance, as the character comparison result of the first text and the second text; wherein the similarity between the strings of the first text and the strings of the second text includes at least one of the following: the similarity ratio of the strings of the first text and the second text compared in sequence, the similarity ratio of the strings of the first text and the second text compared after removing duplicate words, the similarity ratio of the strings of the first text and the second text compared ignoring the order, the similarity ratio of partial strings of the first text and the second text compared, the similarity ratio of partial strings of the first text and the second text compared after removing duplicate words, and the similarity ratio of partial strings of the first text and the second text compared ignoring the order.
[0013] In some embodiments, the method further includes: segmenting the first text and the second text into words respectively; comparing the segmented first text and the segmented second text to determine the word comparison results of the first text and the second text; wherein the word comparison results of the first text and the second text include: the difference in the number of words between the first text and the second text, the ratio of the number of words between the first text and the second text, the difference in the number of words between the first text and the second text after removing duplicate words, the ratio of the number of words between the first text and the second text after removing duplicate words, the difference in the number of words between the first text and the second text after removing duplicate words and stop words, and the comparison results of the number of words between the first text and the second text after removing duplicate words and stop words. The following are at least one of the following: the ratio of the number of words in the second text, the number of repeated words in the first and second texts, the number of repeated words in the first and second texts after removing stop words, the ratio of the number of repeated words in the first and second texts after removing stop words to the total number of words in the first and second texts, the ratio of the importance of repeated words in the first and second texts to the importance of all words, the ratio of the importance of repeated words in the first and second texts after removing stop words to the importance of all words, the Jaccard coefficient of the word set of the first text and the word set of the second text, and whether the first word of the word set of the first text and the first word of the second text are the same.
[0014] In some embodiments, the character comparison result of the first text and the second text further includes at least one of the following: the length difference between the first text and the second text strings, the length difference between the first text and the second text strings after removing stop words, and the length ratio of the first text and the second text strings.
[0015] In some embodiments, the distance between the first text and the second text further includes the minimum word shift distance between the first text and the second text.
[0016] In some embodiments, the comparative statistical features further include at least one of the following: skewness of the text vector of the first text, kurtosis of the text vector of the first text, skewness of the text vector of the second text, and kurtosis of the text vector of the second text.
[0017] In some embodiments, the method further includes: acquiring multiple training sample pairs, each training sample pair including a first sample text and a second sample text; extracting bag-of-words features of the first sample text and the second sample text in each training sample pair respectively; performing a preset first operation on the bag-of-words features of the first sample text and the second sample text to obtain a first sample feature matrix; performing a preset second operation on the bag-of-words features of the first sample text and the second sample text to obtain a second sample feature matrix; training a first logistic regression model based on the first sample feature matrix corresponding to each training sample pair; and training a second logistic regression model based on the second sample feature matrix corresponding to each training sample pair.
[0018] In some embodiments, the method further includes: determining a first similarity of each training sample pair using a trained first logistic regression model, determining a second similarity of each training sample pair using a trained second logistic regression model, and training a classification model based on the first and second similarities of each training sample pair, the first semantic features of the first sample text and the second semantic features of the second sample text in each training sample pair, and the comparative statistical features of the first sample text and the second sample text in each training sample pair.
[0019] In some embodiments, the classification model is an extreme gradient boosting XGBoost model, and the first logistic regression model and the second logistic regression model are connected in parallel and cascaded with the XGBoost model respectively.
[0020] According to some embodiments of this disclosure, a text similarity determination apparatus is provided, comprising: a preliminary similarity determination module, configured to determine the similarity between a first text and a second text based on a first bag-of-words feature of a first text and a second bag-of-words feature of a second text, as a preliminary similarity; a semantic feature determination module, configured to determine a first semantic feature of the first text based on the importance of each word in the first text, and to determine a second semantic feature of the second text based on the importance of each word in the second text; and a similarity determination module, configured to input the preliminary similarity, the first semantic feature, the second semantic feature, and the comparative statistical features of the first text and the second text into a pre-trained classification model to obtain the similarity between the first text and the second text; wherein the comparative statistical features include at least one of the following: the distance between the first text and the second text, the word comparison results of the first text and the second text, and the character comparison results of the first text and the second text.
[0021] According to some other embodiments of this disclosure, a text similarity determination apparatus is provided, comprising: a processor; and a memory coupled to the processor for storing instructions, which, when executed by the processor, cause the processor to perform a text similarity determination method as described in any of the foregoing embodiments.
[0022] According to further embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein the program, when executed by a processor, implements the method for determining text similarity of any of the foregoing embodiments.
[0023] This disclosure determines the preliminary similarity between two texts based on the first bag-of-words features of the first text and the second bag-of-words features of the second text. It then determines the first and second semantic features based on the importance of each word in the first and second texts, respectively. Finally, it obtains comparative statistical features by statistically analyzing at least one of the following: the distance between the first and second texts, the word comparison results, and the character comparison results. Based on the aforementioned preliminary similarity, first semantic features, second semantic features, and comparative statistical features, the similarity between the first and second texts is determined. This disclosure not only applies features extracted separately for the first and second texts but also uses comparative statistical features to determine similarity, enabling a comprehensive comparison of the first and second texts from multiple perspectives and improving the accuracy of text similarity determination.
[0024] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0026] Figure 1 A flowchart illustrating a method for determining text similarity according to some embodiments of this disclosure is shown.
[0027] Figure 2 A schematic diagram illustrating the structure of a text similarity model according to other embodiments of this disclosure is shown.
[0028] Figure 3 A schematic diagram of the structure of a text similarity determination apparatus according to some embodiments of the present disclosure is shown.
[0029] Figure 4A schematic diagram of the structure of a text similarity determination apparatus according to other embodiments of the present disclosure is shown.
[0030] Figure 5 A schematic diagram of the structure of a text similarity determination apparatus according to further embodiments of the present disclosure is shown. Detailed Implementation
[0031] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.
[0032] This disclosure proposes a method for determining text similarity, which is described below in conjunction with... Figure 1 Describe it.
[0033] Figure 1 Flowcharts of some embodiments of the method for determining text similarity in this disclosure. For example... Figure 1 As shown, the method of this embodiment includes steps S102 to S106.
[0034] In step S102, the similarity between the first text and the second text is determined based on the first bag-of-words features of the first text and the second bag-of-words features of the second text, and is used as the preliminary similarity.
[0035] The first bag-of-words features of the first text and the second bag-of-words features of the second text can be determined using the existing bag-of-words model (BOW), which will not be elaborated further here. The first and second bag-of-words features can each be in matrix form. The similarity between the first and second texts can be represented by the Euclidean distance, cosine similarity, or other distance or similarity between the first and second bag-of-words features.
[0036] In some embodiments, a preliminary similarity between the first bag-of-words features and the second bag-of-words features is calculated using a pre-trained logistic regression model. Further, to enrich the features of the two texts, multiple logistic regression models can be used to calculate multiple preliminary similarities between the first bag-of-words features and the second bag-of-words features. For example, a first operation is performed on the first bag-of-words features and the second bag-of-words features to obtain a first feature matrix; a second operation is performed on the first bag-of-words features of the first text and the second bag-of-words features of the second text to obtain a second feature matrix; the first feature matrix is input into a pre-trained first logistic regression model to obtain a first similarity between the output first text and the second text; the second feature matrix is input into a pre-trained second logistic regression model to obtain a second similarity between the output first text and the second text. For example, the first operation is the subtraction of the first bag-of-words features and the second operation is the dot product of the first bag-of-words features and the second bag-of-words features.
[0037] The preset first and second operations are used to fuse the features from the first and second bags of words before inputting them into the logistic regression model. Furthermore, the subtraction and dot product operations themselves help to differentiate the similarity between the two matrices, improving the accuracy of the initial similarity determination. These operations can be replaced with other operations, such as addition, but subtraction and dot product are more effective.
[0038] In step S104, the first semantic features of the first text are determined based on the importance of each word in the first text, and the second semantic features of the second text are determined based on the importance of each word in the second text.
[0039] In some embodiments, the importance of each word in the first text is determined based on the frequency of each word's occurrence in the first text and the frequency of each word's occurrence in the corpus. Alternatively, the importance of each word in the second text is determined based on the frequency of each word's occurrence in the second text and the frequency of each word's occurrence in the corpus. For example, the importance of each word in the first and second texts can be determined by the TF-IDF value of each word, but other methods can also be used to determine the importance of each word, and are not limited to the examples given.
[0040] Furthermore, the importance of each word in the first text is determined, generating a first importance matrix. Similarly, the importance of each word in the second text is determined, generating a second importance matrix. To avoid excessive elements in both matrices, dimensionality reduction can be performed to improve processing efficiency. For example, singular value decomposition (SVD) can be performed on the first importance matrix to obtain a dimensionality-reduced first importance matrix, which serves as the first semantic feature. Similarly, singular value decomposition can be performed on the second importance matrix to obtain a dimensionality-reduced second importance matrix, which serves as the second semantic feature.
[0041] The order of steps S102 and S104 can be interchanged or performed in parallel.
[0042] In step S106, the preliminary similarity, the first semantic feature and the second semantic feature, as well as the comparative statistical features of the first text and the second text are input into the pre-trained classification model to obtain the similarity between the first text and the second text.
[0043] The comparative statistical features may include, for example, at least one of the following: the distance between the first text and the second text, the word comparison results between the first text and the second text, and the character comparison results between the first text and the second text. Furthermore, the comparative statistical features may also include at least one of the following: the skewness of the text vector of the first text, the kurtosis of the text vector of the first text, the skewness of the text vector of the second text, and the kurtosis of the text vector of the second text.
[0044] Comparative statistical characteristics can be predetermined. The following describes in detail various comparative statistical characteristics and their determination methods.
[0045] Regarding the distance between the first and second texts.
[0046] In some embodiments, the first text and the second text are segmented into words respectively; the word vectors of each word in the first text and the word vectors of each word in the second text are determined respectively; the text vector of the first text is determined based on the word vectors of each word in the first text, and the text vector of the second text is determined based on the word vectors of each word in the second text; the spatial distance between the text vectors of the first text and the text vectors of the second text is determined as the distance between the first text and the second text. Spatial distances include, for example, at least one of: Manhattan distance, Jaccard distance, Canberra distance, Euclidean distance, Minkowski distance, and Bray-Curtis distance.
[0047] The word vectors of each word in the first and second texts can be obtained using existing algorithms, such as word embedding algorithms like word2vec and GloVe. The text vector of the first text can be obtained by averaging the word vectors of each word in the first text, and the text vector of the second text can be obtained by averaging the word vectors of each word in the second text. Other algorithms can also be used to determine the text vectors of the first and second texts, not limited to the examples given. To increase the accuracy of text similarity determination, various spatial distances can be used for text similarity determination. The distance between the first and second texts can also include the minimum word move distance (Word Mover's Distance), for details of which can be found in existing technologies and will not be elaborated here.
[0048] The comparison results of words between the first and second texts.
[0049] In some embodiments, the first text and the second text are segmented into words respectively; the segmented first text and the segmented second text are compared to determine the word comparison results of the first text and the second text. The word comparison results of the first text and the second text include, for example: (1) the difference in the number of words in the first text and the second text; (2) the ratio of the number of words in the first text and the second text; (3) the difference in the number of words in the first text and the second text after removing duplicate words; (4) the ratio of the number of words in the first text and the second text after removing duplicate words; (5) the difference in the number of words in the first text and the second text after removing duplicate words and stop words; (6) the ratio of the number of words in the first text and the second text after removing duplicate words and stop words; (7) the number of repeated words in the first text and the second text; (8) the number of words removed from the stop words. (9) The ratio of the number of repeated words in the first and second texts after removing stop words to the total number of words in the first and second texts, (10) The ratio of the importance of repeated words in the first and second texts to the importance of all words, (11) The ratio of the importance of repeated words in the first and second texts to the importance of all words after removing stop words, (12) The Jaccard coefficient of the word set of the first text and the word set of the second text, and (13) Whether the first word of the word set of the first text and the first word of the second text are the same.
[0050] The importance of words can be determined using methods such as TF-IDF. Methods for removing duplicate words and stop words, as well as calculating the Jaccard coefficient between two word sets, can also be achieved using existing techniques, which will not be elaborated here.
[0051] The character comparison results for the first and second texts.
[0052] In some embodiments, the similarity between the strings of the first text and the strings of the second text is determined based on the edit distance, and is used as the character comparison result of the first text and the second text. The similarity between the strings of the first text and the strings of the second text includes, for example, at least one of the following: (1) comparing the similarity ratio of the strings of the first text and the second text in sequence, (2) comparing the similarity ratio of the strings of the first text and the second text after removing duplicate words, (3) comparing the similarity ratio of the strings of the first text and the second text without considering the order, (4) comparing the similarity ratio of partial strings of the first text and the second text, (5) comparing the similarity ratio of partial strings of the first text and the second text after removing duplicate words, and (6) comparing the similarity ratio of partial strings of the first text and the second text without considering the order.
[0053] Determining the similarity between strings by using edit distance (Levenshtein Distance) can employ existing techniques, such as FuzzyWuzzy, a simple and easy-to-use fuzzy string matching toolkit that can be used to obtain various comparison results.
[0054] The character comparison results of the first text and the second text may also include some basic character comparison results, such as at least one of the following: the difference in length between the first text and the second text strings, the difference in length between the first text and the second text strings after removing stop words, and the ratio of the lengths of the first text and the second text strings.
[0055] The aforementioned comparative statistical features, preliminary similarity, first semantic features, and second semantic features are input into a pre-trained classification model to obtain the similarity between the first and second texts. The classification model can be, for example, XGBoost, or other classification models. Due to the large number of comparative statistical features, they can be divided into multiple sets based on feature type. Features such as the distance between the first and second texts, the skewness and kurtosis of the first text's text vector, and the skewness and kurtosis of the second text's text vector can be grouped into the first set; these features reflect the distribution characteristics of the first and second texts. The similarity between the strings of the first and second texts can be grouped into the second set; these features reflect the similarity of the characters in the first and second texts. Furthermore, the remaining word comparison results between the first and second texts, as well as basic character comparison results, such as the difference in string length between the first and second texts, the difference in string length after removing stop words, and the ratio of the string lengths of the first and second texts, can be grouped into a third set. These features reflect the basic feature comparison between the first and second texts.
[0056] The features in the first, second, and third sets can be used to generate different feature vectors or feature matrices in a preset order. These feature vectors or feature matrices, along with the preliminary similarity, the first semantic feature, and the second semantic feature, are then input into a pre-trained classification model to obtain the final similarity between the first and second texts.
[0057] Figure 2 The diagram shown illustrates the text similarity model disclosed herein. Figure 2 As shown, the text similarity model includes: a data input terminal (Data), multiple feature sets (Feature Set), including the first set, second set, and third set in the above embodiment, and a fourth set formed by the first semantic feature and the second semantic feature; a bag-of-words (BoW) model to obtain the first bag-of-words features of the first text and the second bag-of-words features of the second text; two logistic regression models to obtain the first initial similarity and the second initial similarity; and a classification model (e.g., XGBoost) to finally output the similarity between the first text and the second text. Bag-of-words features are sparse data, and the similarity obtained directly through XGBoost is inaccurate. First, the initial similarity of the text is calculated using Logistic Regression, which is then used as input to XGBoost, which helps improve the accuracy of similarity determination.
[0058] In the above embodiments, a first-level model (Logistic Regression) is used to calculate preliminary similarity and concatenated with a second-level model (XGBoost). The input of the second-level model not only includes the preliminary similarity provided by the first-level model but also incorporates a large number of word frequency statistical features. In small samples, semantic features are difficult to train, while statistical features can play a significant role. In large samples, the semantic features of the first-level model combined with the semantic and statistical features of the second-level model result in an overall cascaded model that outperforms previous methods. The model in the above embodiments has strong predictive reliability and low computational cost. Instead of relying on a single model for prediction, the multi-model fusion structure using a second-level model makes the results less susceptible to the influence of anomalous samples compared to other single models. Furthermore, compared to other deep models, this method has lower computational costs.
[0059] The training process for the above model is briefly described below.
[0060] In some embodiments, the logistic regression model and the classification model can be trained separately. Multiple training sample pairs are obtained, each pair including a first sample text and a second sample text. Bag-of-words features of the first sample text and the second sample text are extracted from each training sample pair. A first pre-defined operation is performed on the bag-of-words features of the first and second sample texts to obtain a first sample feature matrix. A second pre-defined operation is performed on the bag-of-words features of the first and second sample texts to obtain a second sample feature matrix. The first logistic regression model is trained based on the first sample feature matrix corresponding to each training sample pair; the second logistic regression model is trained based on the second sample feature matrix corresponding to each training sample pair.
[0061] For example, the first sample feature matrix corresponding to each training sample is input into the first logistic regression model to obtain the output similarity value. The loss function value is calculated based on the output similarity value and the labeled similarity value. The parameters of the first logistic regression model are then readjusted based on the loss function until the stopping condition is met, resulting in the trained first logistic regression model. The second logistic regression model can be trained using a similar method, which will not be elaborated further.
[0062] In some embodiments, a first similarity is determined for each training sample pair using a trained first logistic regression model, and a second similarity is determined for each training sample pair using a trained second logistic regression model. The classification model is trained based on the first and second similarities of each training sample pair, the first semantic features of the first sample text and the second semantic features of the second sample text in each training sample pair, and the comparative statistical features of the first sample text and the second sample text in each training sample pair.
[0063] The comparative statistical features of the first and second sample texts can refer to the aforementioned embodiments, and the types of comparative statistical features used during training are the same as those used in practice. For example, the first and second similarities of each training sample pair, the first semantic features of the first sample text and the second semantic features of the second sample text in each training sample pair, and the comparative statistical features of the first and second sample texts in each training sample pair are input into the classification model to obtain the output similarity value. The loss function value is calculated based on the output similarity value and the labeled similarity value, and the parameters of the classification model are readjusted according to the loss function until the stopping condition is met, resulting in a trained classification model. Alternatively, the following can be used: Figure 2 The entire model shown is trained as a whole, which will not be elaborated on here.
[0064] This disclosure also provides a device for determining text similarity, which is described below in conjunction with... Figure 3 Describe it.
[0065] Figure 3 These are structural diagrams of some embodiments of the apparatus for determining text similarity according to this disclosure. Figure 3 As shown, the device 30 in this embodiment includes: a preliminary similarity determination module 302, a semantic feature determination module 304, and a similarity determination module 306.
[0066] The preliminary similarity determination module 302 is used to determine the similarity between the first text and the second text based on the first bag-of-words features of the first text and the second bag-of-words features of the second text, as a preliminary similarity.
[0067] In some embodiments, the preliminary similarity determination module 302 is used to perform a preset first operation on the first bag-of-words features and the second bag-of-words features to obtain a first feature matrix, wherein the first bag-of-words features and the second bag-of-words features are matrices; to perform a preset second operation on the first bag-of-words features of the first text and the second bag-of-words features of the second text to obtain a second feature matrix; to input the first feature matrix into a pre-trained first logistic regression model to obtain the output first similarity between the first text and the second text; and to input the second feature matrix into a pre-trained second logistic regression model to obtain the output second similarity between the first text and the second text.
[0068] In some embodiments, the first operation is preset to be the subtraction of the first bag-of-words features and the second bag-of-words features; the second operation is preset to be the dot product of the first bag-of-words features and the second bag-of-words features.
[0069] The semantic feature determination module 304 is used to determine the first semantic feature of the first text based on the importance of each word in the first text, and to determine the second semantic feature of the second text based on the importance of each word in the second text.
[0070] In some embodiments, the semantic feature determination module 304 is used to determine the importance of each word in the first text and generate a first importance matrix; perform singular value decomposition on the first importance matrix to obtain a dimension-reduced first importance matrix, which is used as a first semantic feature; or, determine the importance of each word in the second text and generate a second importance matrix; perform singular value decomposition on the second importance matrix to obtain a dimension-reduced second importance matrix, which is used as a second semantic feature.
[0071] In some embodiments, the importance of each word in the first text is determined based on the frequency of each word appearing in the first text and the frequency of each word appearing in the corpus; or, the importance of each word in the second text is determined based on the frequency of each word appearing in the second text and the frequency of each word appearing in the corpus.
[0072] The similarity determination module 306 is used to input the preliminary similarity, the first semantic feature and the second semantic feature, as well as the comparative statistical features of the first text and the second text into the pre-trained classification model to obtain the similarity between the first text and the second text.
[0073] In some embodiments, the comparative statistical features include at least one of the following: the distance between the first text and the second text, the word comparison results of the first text and the second text, and the character comparison results of the first text and the second text.
[0074] In some embodiments, the apparatus further includes: a statistical feature determination module 308, configured to segment the first text and the second text into words respectively; determine the word vectors of each word in the first text and the word vectors of each word in the second text respectively; determine the text vector of the first text based on the word vectors of each word in the first text, and determine the text vector of the second text based on the word vectors of each word in the second text; determine the spatial distance between the text vectors of the first text and the text vectors of the second text, as the distance between the first text and the second text; wherein the spatial distance includes at least one of Manhattan distance, Jaccard distance, Canberra distance, Euclidean distance, Minkowski distance, and Brectis distance.
[0075] In some embodiments, the statistical feature determination module 308 is further configured to determine the similarity between the strings of the first text and the strings of the second text based on the edit distance, as a character comparison result of the first text and the second text. The similarity between the strings of the first text and the strings of the second text includes at least one of the following: the similarity ratio of the strings of the first text and the second text compared in sequence, the similarity ratio of the strings of the first text and the second text after removing duplicate words, the similarity ratio of the strings of the first text and the second text ignoring the order, the similarity ratio of partial strings of the first text and the second text, the similarity ratio of partial strings of the first text and the second text after removing duplicate words, and the similarity ratio of partial strings of the first text and the second text ignoring the order.
[0076] In some embodiments, the statistical feature determination module 308 is further configured to perform word segmentation on the first text and the second text respectively; compare the segmented first text and the segmented second text to determine the comparison result of words in the first text and the second text. The word comparison results of the first text and the second text include at least one of the following: the difference in the number of words in the first text and the second text; the ratio of the number of words in the first text and the second text; the difference in the number of words in the first text and the second text after removing duplicate words; the ratio of the number of words in the first text and the second text after removing duplicate words; the difference in the number of words in the first text and the second text after removing duplicate words and stop words; the ratio of the number of repeated words in the first text and the second text after removing stop words; the ratio of the number of repeated words in the first text and the second text to the total number of words in the first text and the second text after removing stop words; the ratio of the importance of repeated words in the first text and the second text to the importance of all words after removing stop words; the ratio of the importance of repeated words in the first text and the second text to the importance of all words after removing stop words; the Jaccard coefficient of the word set of the first text and the word set of the second text; and whether the first word of the word set of the first text and the second text are the same.
[0077] In some embodiments, the character comparison result of the first text and the second text further includes at least one of the following: the length difference between the first text and the second text strings, the length difference between the first text and the second text strings after removing stop words, and the length ratio of the first text and the second text strings.
[0078] In some embodiments, the distance between the first text and the second text further includes the minimum word shift distance between the first text and the second text.
[0079] In some embodiments, the comparative statistical features further include at least one of the following: skewness of the text vector of the first text, kurtosis of the text vector of the first text, skewness of the text vector of the second text, and kurtosis of the text vector of the second text.
[0080] In some embodiments, the apparatus further includes: a training module 310, configured to acquire multiple training sample pairs, each training sample pair including a first sample text and a second sample text; extract bag-of-words features of the first sample text and the second sample text in each training sample pair respectively; perform a preset first operation on the bag-of-words features of the first sample text and the second sample text to obtain a first sample feature matrix; perform a preset second operation on the bag-of-words features of the first sample text and the second sample text to obtain a second sample feature matrix; train a first logistic regression model based on the first sample feature matrix corresponding to each training sample pair; and train a second logistic regression model based on the second sample feature matrix corresponding to each training sample pair.
[0081] In some embodiments, the training module 310 is further configured to determine the first similarity of each training sample pair using the trained first logistic regression model, and determine the second similarity of each training sample pair using the trained second logistic regression model; and to train the classification model based on the first and second similarities of each training sample pair, the first semantic features of the first sample text and the second semantic features of the second sample text in each training sample pair, and the comparative statistical features of the first sample text and the second sample text in each training sample pair.
[0082] The text similarity determination device in the embodiments of this disclosure can be implemented by various computing devices or computer systems, as described below. Figure 4 as well as Figure 5 Describe it.
[0083] Figure 4 These are structural diagrams of some embodiments of the apparatus for determining text similarity according to this disclosure. Figure 4 As shown, the apparatus 40 of this embodiment includes a memory 410 and a processor 420 coupled to the memory 410. The processor 420 is configured to execute a text similarity determination method in any of the embodiments of this disclosure based on instructions stored in the memory 410.
[0084] The memory 410 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, the operating system, application programs, boot loader, database, and other programs.
[0085] Figure 5 Structural diagrams of other embodiments of the text similarity determination apparatus of this disclosure are shown. Figure 5As shown, the device 50 in this embodiment includes a memory 510 and a processor 520, which are similar to the memory 410 and processor 420, respectively. It may also include an input / output interface 530, a network interface 540, a storage interface 550, etc. These interfaces 530, 540, 550, and the memory 510 and processor 520 can be connected, for example, via a bus 560. The input / output interface 530 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 540 provides a connection interface for various networked devices, such as connecting to a database server or cloud storage server. The storage interface 550 provides a connection interface for external storage devices such as SD cards and USB flash drives.
[0086] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0087] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0090] The above description is only a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.
Claims
1. A method for determining text similarity, comprising: Based on the first bag-of-words features of the first text and the second bag-of-words features of the second text, the similarity between the first text and the second text is determined as a preliminary similarity. Determining a first semantic feature of the first text based on the importance of each word in the first text includes: determining the importance of each word in the first text and generating a first importance matrix; performing singular value decomposition on the first importance matrix to obtain a dimension-reduced first importance matrix, which is used as the first semantic feature. Determining a second semantic feature of the second text based on the importance of each word in the second text includes: determining the importance of each word in the second text and generating a second importance matrix; performing singular value decomposition on the second importance matrix to obtain a dimension-reduced second importance matrix, which is used as the second semantic feature. The preliminary similarity, the first semantic feature, the second semantic feature, and the comparative statistical features of the first and second texts are input into a pre-trained classification model to obtain the similarity between the first and second texts. The comparative statistical features include at least one of the following: the distance between the first text and the second text, the word comparison results between the first text and the second text, and the character comparison results between the first text and the second text.
2. The determination method according to claim 1, wherein, Determining the similarity between the first text and the second text based on the first bag-of-words features of the first text and the second bag-of-words features of the second text includes: Perform a preset first operation on the first bag-of-words features and the second bag-of-words features to obtain a first feature matrix, wherein the first bag-of-words features and the second bag-of-words features are matrices; Perform a preset second operation on the first bag-of-words features of the first text and the second bag-of-words features of the second text to obtain a second feature matrix; The first feature matrix is input into a pre-trained first logistic regression model to obtain the first similarity between the first text and the second text. The second feature matrix is input into a pre-trained second logistic regression model to obtain the output second similarity between the first text and the second text.
3. The determination method according to claim 2, wherein, The preset first operation is the subtraction of the first bag-of-words feature and the second bag-of-words feature; The preset second operation is the dot product of the first bag-of-words feature and the second bag-of-words feature.
4. The determination method according to claim 1, wherein, The importance of each word in the first text is determined based on the frequency of each word in the first text and the frequency of each word in the corpus; Alternatively, the importance of each word in the second text can be determined based on the frequency of each word's occurrence in the second text and the frequency of each word's occurrence in the corpus.
5. The determining method according to claim 1, further comprising: Perform word segmentation on the first text and the second text respectively; Determine the word vectors of each word in the first text and the word vectors of each word in the second text respectively; The text vector of the first text is determined based on the word vectors of each word in the first text, and the text vector of the second text is determined based on the word vectors of each word in the second text. The spatial distance between the text vector of the first text and the text vector of the second text is determined as the distance between the first text and the second text. The spatial distances include at least one of the following: Manhattan distance, Jaccard distance, Canberra distance, Euclidean distance, Minkowski distance, and Brectis distance.
6. The determining method according to claim 1, further comprising: The similarity between the strings of the first text and the strings of the second text is determined based on the edit distance, and this is used as the character comparison result between the first text and the second text. The similarity between the strings of the first text and the strings of the second text includes at least one of the following: the similarity ratio of the strings of the first text and the second text compared in sequence, the similarity ratio of the strings of the first text and the second text after removing duplicate words, the similarity ratio of the strings of the first text and the second text ignoring the order, the similarity ratio of partial strings of the first text and the second text compared, the similarity ratio of partial strings of the first text and the second text after removing duplicate words, and the similarity ratio of partial strings of the first text and the second text ignoring the order.
7. The determining method according to claim 1, further comprising: Perform word segmentation on the first text and the second text respectively; The first text after word segmentation and the second text after word segmentation are compared to determine the word comparison results of the first text and the second text; The word comparison results between the first text and the second text include: the difference in the number of words between the first text and the second text; the ratio of the number of words between the first text and the second text; the difference in the number of words between the first text and the second text after removing duplicate words; the ratio of the number of words between the first text and the second text after removing duplicate words; the difference in the number of words between the first text and the second text after removing duplicate words and stop words; the ratio of the number of words between the first text and the second text after removing duplicate words and stop words; the number of repeated words in the first text and the second text; and the number of stop words removed. The following criteria are considered: the number of repeated words in the first and second texts; the ratio of the number of repeated words in the first and second texts after removing stop words to the total number of words in the first and second texts; the ratio of the importance of repeated words in the first and second texts to the importance of all words; the ratio of the importance of repeated words in the first and second texts after removing stop words to the importance of all words; the Jaccard coefficient of the word sets of the first and second texts; and whether the first word of the word sets of the first and second texts is the same.
8. The determining method according to claim 6, wherein, The character comparison results between the first text and the second text also include at least one of the following: the length difference between the first text and the second text strings, the length difference between the first text and the second text strings after removing stop words, and the length ratio between the first text and the second text strings.
9. The determining method according to claim 5, wherein, The distance between the first text and the second text also includes the minimum word shift distance between the first text and the second text.
10. The determination method according to claim 1, wherein, The comparative statistical features also include at least one of the following: skewness of the text vector of the first text, kurtosis of the text vector of the first text, skewness of the text vector of the second text, and kurtosis of the text vector of the second text.
11. The determining method according to claim 2, further comprising: Obtain multiple training sample pairs, each training sample pair including a first sample text and a second sample text; Extract the bag-of-words features of the first sample text and the second sample text from each training sample pair respectively, and perform a preset first operation on the bag-of-words features of the first sample text and the bag-of-words features of the second sample text to obtain the first sample feature matrix; The bag-of-words features of the first sample text and the bag-of-words features of the second sample text are subjected to a preset second operation to obtain the second sample feature matrix; The first logistic regression model is trained based on the feature matrix of the corresponding first sample for each training sample. The second logistic regression model is trained based on the feature matrix of the corresponding second sample for each training sample.
12. The determining method according to claim 11, further comprising: The first similarity of each training sample pair is determined using the first logistic regression model that has been trained, and the second similarity of each training sample pair is determined using the second logistic regression model that has been trained. The classification model is trained based on the first and second similarities of each training sample pair, the first semantic features of the first sample text and the second semantic features of the second sample text in each training sample pair, and the comparative statistical features of the first sample text and the second sample text in each training sample pair.
13. The determining method according to claim 2, wherein, The classification model is an extreme gradient boosting XGBoost model, and the first logistic regression model and the second logistic regression model are connected in parallel and respectively cascaded with the XGBoost model.
14. A device for determining text similarity, comprising: The preliminary similarity determination module is used to determine the similarity between the first text and the second text based on the first bag-of-words features of the first text and the second bag-of-words features of the second text, as the preliminary similarity. A semantic feature determination module is used to determine a first semantic feature of the first text based on the importance of each word in the first text, including: determining the importance of each word in the first text and generating a first importance matrix; performing singular value decomposition on the first importance matrix to obtain a dimension-reduced first importance matrix, which is used as the first semantic feature; and determining a second semantic feature of the second text based on the importance of each word in the second text, including: determining the importance of each word in the second text and generating a second importance matrix; performing singular value decomposition on the second importance matrix to obtain a dimension-reduced second importance matrix, which is used as the second semantic feature. The similarity determination module is used to input the preliminary similarity, the first semantic feature and the second semantic feature, and the comparative statistical features of the first text and the second text into a pre-trained classification model to obtain the similarity between the first text and the second text. The comparative statistical features include at least one of the following: the distance between the first text and the second text, the word comparison results between the first text and the second text, and the character comparison results between the first text and the second text.
15. A device for determining text similarity, comprising: processor; as well as A memory coupled to the processor is used to store instructions that, when executed by the processor, cause the processor to perform the method for determining text similarity as described in any one of claims 1-13.
16. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the program implements the steps of the method according to any one of claims 1-13.