Text similarity determination method, device, equipment and storage medium

Abstract

The embodiment of the present invention discloses a method, device, equipment and storage medium for determining text similarity. The method comprises: obtaining a target text and an alternative text whose similarity is to be determined; determining the word semantic similarity and word frequency inverse word frequency similarity between the target text and the alternative text, wherein the word frequency inverse word frequency similarity includes the word frequency inverse word frequency similarity without part of speech and / or the word frequency inverse word frequency similarity with part of speech; determining the text similarity between the target text and the alternative text based on a preset word semantic weight, a preset word frequency inverse word frequency weight, and the word semantic similarity and the word frequency inverse word frequency similarity, wherein the preset word frequency inverse word frequency weight includes the preset word frequency inverse word frequency weight without part of speech and / or the preset word frequency inverse word frequency weight with part of speech. Through the above technical solution, a more accurate determination of the text similarity is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to big data mining technology, and more particularly to a text similarity determination method, apparatus, device and storage medium. Background Art

[0002] With the development of the Internet, more and more data appears on the Internet in the form of text, such as microblog messages, news headlines, forum comments, product reviews on e-commerce platforms, product questions and buyers' answers to product questions, etc. Applying machine learning technology to Internet text data to mine valuable information to provide useful convenience for people's lives and meet different needs has become a very hot topic in big data application technology.

[0003] Take the questions and answers about products on e-commerce platforms as an example. When shoppers make a purchase decision, they usually ask questions to people who have purchased the product, or browse the Q&A data under the product, or ask customer service to fully understand the real information of the product. From the user's perspective, the user needs to use the questions he wants to ask as a benchmark, browse the historical Q&A data, and see if anyone has asked similar questions. If the number of questions is large, it may be difficult to find a satisfactory answer; from the customer service perspective, it is necessary to calculate the similarity between the questions raised by the user and the questions already in the question library, find the most similar questions, and use the answers to similar questions to answer the user's questions. In this way, it is necessary to calculate the similarity between the user's questions and the questions already in the question library.

[0004] At present, the solution for determining the similarity of texts for problems such as the above mainly adopts the vector space model, that is, mapping each word in the text into a vector space, and calculating the cosine distance between the vectors. The smaller the distance, the greater the similarity of the words. There are two main similarity determination schemes based on the vector space model. One is the similarity determination scheme based on word importance, such as the Term Frequency-Inverse Document Frequency (TF-IDF) model. TF-IDF is used to evaluate the importance of a word to a document in a text library. The importance of the word increases in direct proportion to the number of times it appears in the document, but at the same time it decreases in inverse proportion to the frequency of its appearance in the corpus. The process of the similarity determination (i.e., problem matching) scheme based on TF-IDF is as follows: Figure 1As shown. The typical application of this solution is in a search engine, where the user inputs a search question, the search question is segmented, a mapping relationship between the segmented words and documents is established, and after finding all the documents, the similarity score and ranking between the user's search question and the existing documents in the knowledge base are calculated based on the similarity algorithm (i.e., TF / IDF algorithm), and the results are returned according to the document ranking. The other is a similarity determination solution based on natural language processing (Natural Language Processing, NLP), and the commonly used model is the Word2vec model. The Word2vec model is trained based on Harris's distribution hypothesis (i.e., words with similar contexts are also similar in semantics). The neural network model obtained through training can map each word to a K-dimensional space to represent the word with a vector, and then measure the semantic similarity of the word through the similarity between vectors in the vector space. The solution flow of the semantic similarity model based on NLP is as follows. Figure 2 shown.

[0005] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: 1) The similarity determination scheme based on TF-IDF only measures the importance of a word to a document by the inverse of word frequency, without considering the position information of the word or the semantic information of the word. The similarity between texts determined by it is of low accuracy and cannot meet the practical application requirements such as finding the most similar texts in question-answering systems. 2) The similarity determination scheme based on NLP is only effective in similarity calculation at the word level. When extended to similarity calculation at the sentence level, due to the complexity of the syntactic structure, the sentence vector is simply represented by accumulating or concatenating the word vectors of all words in the sentence, which cannot achieve the ideal effect in actual application. Summary of the invention

[0006] Embodiments of the present invention provide a method, apparatus, device and storage medium for determining text similarity, so as to more accurately determine the similarity of texts.

[0007] In a first aspect, an embodiment of the present invention provides a method for determining text similarity, comprising:

[0008] Obtaining a target text and an alternative text whose similarity is to be determined;

[0009] Determine the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text, wherein the word frequency inverse word frequency similarity includes word frequency inverse word frequency similarity without part of speech and / or word frequency inverse word frequency similarity with part of speech;

[0010] The text similarity between the target text and the alternative text is determined based on the preset word semantic weight, the preset word frequency inverse word frequency weight, the word semantic similarity and the word frequency inverse word frequency similarity, wherein the preset word frequency inverse word frequency weight includes the preset part-of-speech word frequency inverse word frequency weight and / or the preset part-of-speech word frequency inverse word frequency weight.

[0011] In a second aspect, an embodiment of the present invention further provides a device for determining text similarity, the device comprising:

[0012] A target text acquisition module, used to acquire a target text and an alternative text whose similarity is to be determined;

[0013] A first similarity determination module is used to determine the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text, wherein the word frequency inverse word frequency similarity includes word frequency inverse word frequency similarity without part of speech and / or word frequency inverse word frequency similarity with part of speech;

[0014] The second similarity determination module is used to determine the text similarity between the target text and the alternative text based on a preset word semantic weight, a preset word frequency inverse word frequency weight, and the word semantic similarity and the word frequency inverse word frequency similarity, wherein the preset word frequency inverse word frequency weight includes a preset part-of-speech word frequency inverse word frequency weight and / or a preset part-of-speech word frequency inverse word frequency weight.

[0015] In a third aspect, an embodiment of the present invention further provides a device, the device comprising:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the text similarity determination method provided by any embodiment of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the text similarity determination method provided by any embodiment of the present invention.

[0020] The embodiment of the present invention determines the text similarity between the target text and the candidate text by generating the word semantic similarity and the word frequency inverse frequency similarity including the inverse frequency similarity of the word frequency without part of speech and / or the inverse frequency similarity of the word frequency with part of speech; according to the preset word semantic weight, the preset word frequency inverse frequency weight including the preset inverse frequency weight of the word frequency without part of speech and / or the preset inverse frequency weight of the word frequency with part of speech, and the word semantic similarity and the word frequency inverse frequency similarity. It realizes the use of word semantics and multiple dimensional features of word frequency and / or part of speech to grasp the overall semantics of the sentence of the text, thereby characterizing the text similarity from different dimensions, obtaining the comprehensive similarity of the text, and greatly improving the accuracy of the text similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of a similarity determination method based on a word frequency inverse word frequency model in the prior art;

[0022] Figure 2 It is a flow chart of a similarity determination method based on a semantic similarity model of NLP in the prior art;

[0023] Figure 3a is a flow chart of a method for determining text similarity in Embodiment 1 of the present invention;

[0024] Figure 3b It is a logical framework diagram of a method for determining text similarity in Embodiment 1 of the present invention;

[0025] Figure 4a is a flow chart of a method for determining text similarity in Embodiment 2 of the present invention;

[0026] Figure 4b is a schematic diagram of the CBOW model structure in the second embodiment of the present invention;

[0027] Figure 4c is a schematic diagram of the Skip-Gram model structure in the second embodiment of the present invention;

[0028] Figure 5 is a flow chart of a method for determining text similarity in Embodiment 3 of the present invention;

[0029] Figure 6a is a flow chart of a method for determining text similarity in Embodiment 4 of the present invention;

[0030] Figure 6b It is a logical framework diagram of a method for determining text similarity in Embodiment 4 of the present invention;

[0031] Figure 7 is a structural schematic diagram of a device for determining text similarity provided by Embodiment 5 of the present invention;

[0032] Figure 8 It is a structural diagram of a device in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0034] Embodiment 1

[0035] The text similarity determination method provided in this embodiment can be applied to the similarity calculation of topics, messages and replies, consultations, suggestions and feedback in online forums, network-based intelligent questions and answers, instant chat records, etc. The method can be executed by a text similarity determination device, which can be implemented in software and / or hardware. The device can be integrated in a device with large-scale data computing functions, such as a personal computer or server. The embodiment of the present invention is described by taking intelligent questions and answers as an example. See Figure 3a The method of this embodiment specifically includes the following steps:

[0036] S110: Obtain the target text and the candidate texts whose similarities are to be determined.

[0037] The target text is the text for which similarity needs to be calculated. It can be an existing text or a new text obtained from the outside. The candidate text is the text used to calculate similarity with the target text. The text here can be a short text or a long text. Short text refers to a short text such as a sentence or a few short sentences (a small paragraph).

[0038] In specific implementation, the content input by the user can be received as the target text. At the same time, the candidate text is obtained from the existing text that can be collected, and the candidate text can be one or more. It should be understood that a text database can be constructed in advance based on the existing text that can be collected, and then the candidate text is obtained from the text database.

[0039] S120: Determine the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text.

[0040] Among them, word semantic similarity is the similarity determined from the semantic dimension of the word. Word frequency inverse word frequency similarity is the similarity determined from the dimension of word importance. The measurement basis of word frequency similarity is whether there are the same words with similar frequencies in two sentences. If so, they are similar. Inverse word frequency measures the importance of each word to the sentence, mainly based on the fact that the more times the word appears in the sentence and the fewer times it appears in other sentences, the more important the word is to the sentence. Word frequency and inverse word frequency jointly characterize the characteristics of each word in the sentence, so as to calculate the similarity of the sentences.

[0041] Exemplarily, the word frequency inverse word frequency similarity includes the word frequency inverse word frequency similarity without part of speech and / or the word frequency inverse word frequency similarity with part of speech. The word frequency inverse word frequency similarity without part of speech refers to the word frequency inverse word frequency similarity that does not distinguish the part of speech of each word in the target text, while the word frequency inverse word frequency similarity with part of speech refers to the word frequency inverse word frequency similarity that distinguishes the part of speech of each word in the target text. Since the part of speech can reflect the semantics of the word to a certain extent, the word frequency inverse word frequency similarity with part of speech can be understood as a similarity determined from two dimensions: the semantics of the word and the importance of the word.

[0042] In the related art, similarity can only be determined from one dimension of words. When faced with text similarity calculations with many words and more sentence semantics, the methods for determining similarity in the related art cannot accurately determine the similarity of the text. Therefore, in the embodiment of the present invention, at least two dimensions of similarity are used to determine text similarity. Specifically, the word semantic dimension and the word importance dimension are used. The word importance dimension can be divided into a word importance dimension without part of speech and a word importance dimension with part of speech. Although the text similarity is still measured from the granularity of the word, the similarity of the text can be more completely represented by fusing the similarities of multiple dimensions.

[0043] In specific implementation, the target text and the alternative texts can be used as inputs of a word semantic similarity model for calculating word semantic similarity. Through the similarity calculation of the word semantic similarity model, the word semantic similarity between the target text and the alternative text can be obtained. Similarly, the target text and the alternative text can be used as inputs of a word frequency inverse word frequency similarity model for calculating word frequency inverse word frequency similarity. Through the similarity calculation of the word frequency inverse word frequency similarity model, the word frequency inverse word frequency similarity between the target text and the alternative text can be obtained. The above-mentioned word semantic similarity model can be a similarity model based on NLP, for example, it can be a word vector learning model such as a Word2vec model, a Glove model or a Bert model. Because the above-mentioned word frequency inverse word frequency similarity model needs to calculate the inverse word frequency similarity of word frequency without part of speech and the inverse word frequency similarity of word frequency with part of speech, the word frequency inverse word frequency similarity model can be divided into a word frequency inverse word frequency similarity model without part of speech and a word frequency inverse word frequency similarity model with part of speech. These two word frequency inverse word frequency similarity models can be the same model, for example, it can be a TF-IDF model, but the input data is different, for example, one inputs a phrase that does not distinguish between parts of speech, and the other inputs a phrase that distinguishes between parts of speech; or they can be two different models, in which case the word frequency inverse word frequency similarity model with part of speech is required to perform specific processing on part of speech.

[0044] The above process calculates the similarity of a single candidate text. If there are multiple candidate texts, multiple cycles of the above process are required to complete the similarity calculation between all candidate texts and the target text. In order to increase the calculation speed and thus improve the efficiency of determining the candidate text that is more similar to the target text among multiple candidate texts, all candidate texts can be calculated in parallel. For example, each candidate text can be represented in advance as a vector of different dimensions, so that all candidate texts can be represented in different matrix forms. For specific implementation, this process can be referred to Figure 3b , all candidate texts are pre-represented as word semantic feature matrices and word frequency inverse frequency feature matrices (including word frequency inverse frequency feature matrices without part of speech and / or word frequency inverse frequency feature matrices with part of speech). This process is called stock calculation. After the target text is determined, the target text is represented as word semantic feature vectors and word frequency inverse frequency feature vectors (including word frequency inverse frequency feature vectors without part of speech and / or word frequency inverse frequency feature vectors with part of speech) based on the same model as the stock calculation. This process is called real-time incremental calculation. Finally, the word semantic feature vectors and word semantic feature matrices can be used to calculate word semantic similarity, and the word frequency inverse frequency feature vectors and word frequency inverse frequency feature matrices can be used to calculate word frequency inverse frequency similarity, so as to determine the word semantic similarity and word frequency inverse frequency similarity between the target text and each candidate text.

[0045] It should be noted that the inverse frequency similarity of word frequency without part of speech and the inverse frequency similarity of word frequency with part of speech do not have to be calculated in full, but only one of them can be selected. In other words, one or two of the inverse frequency similarity model of word frequency without part of speech and the inverse frequency similarity model of word frequency with part of speech may need to be used in this operation. At this time, the similarity determined in this operation can include word semantic similarity and inverse frequency similarity of word frequency without part of speech, word semantic similarity and inverse frequency similarity of word frequency with part of speech, as well as word semantic similarity, inverse frequency similarity of word frequency without part of speech and inverse frequency similarity of word frequency with part of speech.

[0046] S130, determining the text similarity between the target text and the candidate texts according to the preset word semantic weight, the preset word frequency inverse word frequency weight, the word semantic similarity and the word frequency inverse word frequency similarity.

[0047] Among them, the preset word semantic weight and the preset word frequency inverse word frequency weight both refer to the preset weight values, which are used to determine the proportion of word semantic similarity and word frequency inverse word frequency similarity in the similarity fusion process, respectively, which can be set in advance by human experience. Text similarity refers to the similarity after the similarities of different dimensions are fused, which can represent the comprehensive similarity between the target text and the candidate text.

[0048] Exemplarily, the preset word frequency inverse word frequency weight includes a preset non-part-of-speech word frequency inverse word frequency weight and / or a preset part-of-speech word frequency inverse word frequency weight. It should be noted that the preset word frequency inverse word frequency weight corresponds to the word frequency inverse word frequency similarity. If the word frequency inverse word frequency similarity is the non-part-of-speech word frequency inverse word frequency similarity, then the preset word frequency inverse word frequency weight is the preset non-part-of-speech word frequency inverse word frequency weight; if the word frequency inverse word frequency similarity is the part-of-speech word frequency inverse word frequency similarity, then the preset word frequency inverse word frequency weight is the preset part-of-speech word frequency inverse word frequency weight; if the word frequency inverse word frequency similarity is the part-of-speech word frequency inverse word frequency similarity and the part-of-speech word frequency inverse word frequency similarity, then the preset word frequency inverse word frequency weight is the preset non-part-of-speech word frequency inverse word frequency weight and the preset part-of-speech word frequency inverse word frequency weight.

[0049] After obtaining the similarity between the target text and the candidate text in different dimensions, it is necessary to fuse the similarities in each dimension to obtain multi-dimensional text similarity. In specific implementation, according to the weight of the similarity in each dimension, the similarities in different dimensions are weighted summed to determine the text similarity. The weighted summation formula for determining the text similarity is as shown in formula (1):

[0050] score=w 1 score 1 +w 2 score 2 (1)

[0051] Among them, score represents the text similarity between the target text and the alternative text, w 1 and w 2 Respectively represent the preset word semantic weight and the preset word frequency inverse word frequency weight, score 1 and score 2 They represent word semantic similarity and word frequency inverse frequency similarity respectively.

[0052] The technical solution of this embodiment determines the text similarity between the target text and the candidate text by generating the word semantic similarity and the word frequency inverse frequency similarity including the inverse frequency similarity of the word frequency without part of speech and / or the inverse frequency similarity of the word frequency with part of speech; according to the preset word semantic weight, the preset word frequency inverse frequency weight including the preset inverse frequency weight of the word frequency without part of speech and / or the preset inverse frequency weight of the word frequency with part of speech, and the word semantic similarity and the word frequency inverse frequency similarity. It realizes the use of multiple dimensional features of word semantics and word frequency and / or part of speech to grasp the overall semantics of the sentence of the text, thereby characterizing the text similarity from different dimensions, obtaining the comprehensive similarity of the text, and greatly improving the accuracy of the text similarity.

[0053] Embodiment 2

[0054] Based on the above-mentioned embodiment 1, this embodiment further optimizes "determining the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text", and optimizes "determining the text similarity between the target text and the candidate text according to the preset word semantic weight, the preset word frequency inverse word frequency weight, and the word semantic similarity and word frequency inverse word frequency similarity". The explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here. Figure 4a , the text similarity determination method provided in this embodiment includes:

[0055] S210: Obtain the target text and the candidate texts whose similarities are to be determined.

[0056] S220 , segment the target text to obtain target part-of-speech words corresponding to the target text.

[0057] Since the target text contains at least one sentence, and the similarity is determined at the word level, the target text needs to be segmented first so as to split the target text into multiple words. Since each word after segmentation has no part of speech, each word obtained by segmentation is the target part-of-speech-free word corresponding to the target text.

[0058] S230 , determining the word semantic similarity between the target text and the candidate text based on each target part-of-speech word and the candidate text and based on a pre-trained word semantic similarity model.

[0059] According to the description of the above embodiment, the word semantic similarity model can be a Word2vec model, which uses a shallow two-layer neural network to train the text by inputting the context of the target word to predict the target word (corresponding to the CBOW model structure), or inputting the target word to predict the context of the word (corresponding to the Skip-Gram model structure), and obtain the parameters of the network hidden layer, that is, obtain the trained Word2vec model. The trained Word2vec model can map each word to a vector space and represent the word as a corresponding feature vector.

[0060] See also Figure 4b , the CBOW model has three layers, namely the input layer, the hidden layer and the output layer, which is the prediction P(w t |w t-k ,w t-(k-1) ,L,w t-1 ,w t+1 ,w t+2 ,L,w t+k ), where w t is the target word to be predicted, w t-k ,w t-(k-1) ,L,w t-1 ,w t+1 ,w t+2 ,L,w t+k (k=2) is the context of the target word, that is, the first two words and the last two words of the target word are selected as the context. The operation from the input layer to the hidden layer is the sum of the context vectors, and the hidden layer to the output layer uses the sub-level Softmax or negative sampling (Negative Simpling). Figure 4c The Skip-Gram model also has three layers, namely the input layer, the hidden layer, and the output layer. However, in contrast to the CBOW model, the Skip-Gram model predicts P(w i |w t ), where tc≤i≤t+c and i≠t, c is the window size (a constant representing the context size), assuming there is a w 1 ,w 2 ,w 3 ,…,w T The goal of Skip-gram is to maximize:

[0061]

[0062]

[0063] After determining the Word2vec model structure, model training is required. The model training process can be found in Figure 3b The inventory calculation process.

[0064] Exemplarily, the Word2vec model is pre-trained based on a short text database and a long text database, wherein the long text database is pre-constructed based on business data corresponding to a business scenario, and the short text database is pre-constructed based on business data corresponding to business needs in the business scenario.

[0065] Among them, the short text database refers to a database composed of a large number of short texts, and the source of the collection of short text data in the short text database is related to specific business needs. For example, if the business requirement is only to calculate similarity, then the data source of the short text database can be short texts in any network platform, and the more sources the better; if the business requirement is to determine the reference answer based on similar questions, then because the focus of the answers in different intelligent question-and-answer systems is different, the data source of the short text database can only be the questions asked in the intelligent question-and-answer system corresponding to the target text. The long text database refers to texts with more sentences, such as articles or product manuals, and the source of its data collection is related to specific business needs. For example, if the business scenario is intelligent question-and-answer, then the long text database can be constructed based on long texts with stronger sentence logic, such as expert recommendation articles, product introductions, or product manuals, collected from any intelligent question-and-answer system.

[0066] Since the target text can be either short text or long text, in order to enhance the text compatibility of the Word2vec model, this embodiment uses both short text database and long text database for model training, and in order to improve the semantic expression of the Word2vec model, the more complete the words in the long text database, the better the training effect of the Word2vec model. In the specific implementation, first of all, it is necessary to obtain training data, that is, to obtain long text database and short text database according to business scenarios and business needs. Then, preprocessing such as word segmentation and data cleaning is performed on it. Data cleaning is used to remove some stop words and punctuation marks and retain only valid data. Afterwards, the preprocessed long text database and short text database are input into the Word2vec model for model training to obtain the trained Word2vec model.

[0067] In the incremental calculation part, each target word without part of speech is input into the trained word semantic similarity model Word2vec model to obtain the row vector representation of each target word without part of speech. After that, the mean of the corresponding columns of multiple row vectors is calculated to obtain a row vector of the column mean as the word semantic feature vector of the target text. Similarly, the word semantic feature vector corresponding to the candidate text can be obtained. Finally, the vector cosine of the word semantic feature vector of the target text and the word semantic feature vector corresponding to the candidate text is calculated to obtain the word semantic similarity between the target text and the candidate text.

[0068] Exemplarily, S230 includes: inputting each target part-of-speech word into a word semantic similarity model to generate a word semantic feature vector corresponding to the target text; determining the word semantic similarity between the target text and the alternative text corresponding to the row vector based on the word semantic feature vector and the row vector in the word semantic feature matrix; wherein the word semantic feature matrix is ​​generated based on the part-of-speech word segmentation results of the text database and the word semantic similarity model.

[0069] When each text in the text database is an alternative text, in order to improve the operation efficiency, each alternative text in the text database can be pre-processed by word segmentation and data cleaning to obtain the corresponding word segmentation results without part of speech. Then it is input into the Word2vec model to obtain the word semantic feature matrix of the text database. Each row vector in the word semantic feature matrix represents the feature vector of an alternative text. Finally, the vector cosine between the word semantic feature vector and each row vector in the word semantic feature matrix is ​​calculated respectively, and the word semantic similarity between the target text and each alternative text in the text database can be obtained.

[0070] S240 , according to each target part-of-speech word, the target text and the candidate text, based on a word frequency inverse word frequency similarity model, determine the part-of-speech word frequency inverse word frequency similarity between the target text and the candidate text.

[0071] The term frequency-inverse term frequency similarity model in this embodiment adopts the term frequency-inverse term frequency model TF-IDF model, which is constructed based on the term frequency TF value and the inverse term frequency IDF value of the word. Term frequency (TF) is the frequency of a given word in a text. The calculation of the TF value is to normalize the term frequency to prevent long texts. Inverse document frequency (IDF) is a measure of the general importance of a word. The specific calculation formula is as follows:

[0072]

[0073]

[0074]

[0075] Among them, n i,j Representation word t i In text d j The number of occurrences in k n k,j Represents text d j The sum of the number of occurrences of all words in the text database, |D| is the total number of texts in the text database, |{j:t i ∈d j}| indicates that the text database contains word t iTo ensure that the denominator is not 0, we usually use |{j:t i ∈d j}|+1.

[0076] According to the calculation formula of the above TF-IDF value, it is necessary to train the word frequency inverse frequency model based on the text database in advance to determine |D|. At the same time, it is necessary to determine the vector dimension of the vector representation of a word based on the number of non-repeated words in the text database. After obtaining the trained TF-IDF model, each target part-of-speech word, the target text and the text database can be input into the TF-IDF model to obtain the TF-IDF value of each target part-of-speech word to form the part-of-speech word frequency inverse frequency feature vector of the target text. The number of columns of this vector is consistent with the vector dimension determined above, in which the TF-IDF value is filled in the corresponding element position of the target part-of-speech word, and the remaining element positions are filled with 0. Similarly, the part-of-speech word frequency inverse frequency feature vector corresponding to the alternative text can be obtained. Finally, the vector cosine of the part-of-speech word frequency inverse frequency feature vector of the target text and the part-of-speech word frequency inverse frequency feature vector corresponding to the alternative text is calculated to obtain the part-of-speech word frequency inverse frequency similarity between the target text and the alternative text.

[0077] Exemplarily, S240 includes: inputting each target part-of-speech word and the target text into a word frequency inverse word frequency similarity model to generate a part-of-speech word frequency inverse word frequency feature vector corresponding to the target text; determining the part-of-speech word frequency inverse word frequency similarity between the target text and the alternative text corresponding to the row vector based on the part-of-speech word frequency inverse word frequency feature vector and the row vector in the part-of-speech word frequency inverse word frequency feature matrix; wherein the part-of-speech word frequency inverse word frequency feature matrix is ​​generated based on the part-of-speech word segmentation result of the text database and the word frequency inverse word frequency similarity model.

[0078] When each text in the text database is an alternative text, in order to improve the operation efficiency, each alternative text in the text database can be pre-processed by word segmentation and data cleaning to obtain the corresponding part-of-speech word segmentation results. Then input it into the TF-IDF model to obtain the part-of-speech word frequency inverse word frequency feature matrix of the text database. The number of columns of the part-of-speech word frequency inverse word frequency feature matrix is ​​consistent with the above vector dimension. The element position without corresponding part-of-speech words is filled with 0, and each row vector in the matrix represents the feature vector of an alternative text. Finally, the vector cosine between the part-of-speech word frequency inverse word frequency feature vector and each row vector in the part-of-speech word frequency inverse word frequency feature matrix is ​​calculated respectively, and the part-of-speech word frequency inverse word frequency similarity between the target text and each alternative text can be obtained.

[0079] S250, determining the text similarity between the target text and the candidate texts according to the preset word semantic weight, the preset inverse word frequency weight of the word without part of speech, the word semantic similarity and the inverse word frequency similarity of the word without part of speech.

[0080] The preset word frequency inverse word frequency weight and word frequency inverse word frequency similarity in formula (1) are respectively determined as the preset part-of-speech word frequency inverse word frequency weight w 21 Similarity score with the inverse frequency of the word without part of speech 21 , the text similarity between the target text and the candidate text can be obtained according to the following formula:

[0081] score=w 1 score 1 +w 21 score 21 (7)

[0082] The technical solution of this embodiment determines the word semantic similarity and the inverse frequency similarity of part-of-speech words between the target text and the alternative texts, and determines the text similarity between the target text and the alternative texts based on preset word semantic weights, preset inverse frequency weights of part-of-speech words, and word semantic similarity and inverse frequency similarity of part-of-speech words, thereby realizing the determination of the text similarity of the target text from two dimensions of word semantics and importance of part-of-speech words, and improving the accuracy of determining text similarity to a certain extent.

[0083] Embodiment 3

[0084] Based on the above-mentioned embodiment 1, this embodiment further optimizes "determining the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text", and optimizes "determining the text similarity between the target text and the candidate text according to the preset word semantic weight, the preset word frequency inverse word frequency weight, and the word semantic similarity and word frequency inverse word frequency similarity". The explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here. Figure 5 , the text similarity determination method provided in this embodiment includes:

[0085] S310: Obtain the target text and the candidate texts whose similarities are to be determined.

[0086] S320: Segment and tag the target text to obtain target part-of-speech words and target part-of-speech tagged words corresponding to the target text.

[0087] In this embodiment, the part of speech of each word needs to be distinguished, so after the target text is segmented, each target part-of-speech word obtained needs to be tagged with a part of speech to obtain a target part-of-speech tagged word. If a target part-of-speech word has two or more parts of speech, then the target part-of-speech word generates two or more target part-of-speech tagged words.

[0088] S330 , determining the word semantic similarity between the target text and the candidate text based on each target part-of-speech word and the candidate text and based on a pre-trained word semantic similarity model.

[0089] S340, according to each target part-of-speech tagged word, the target text and the candidate text, based on the word frequency inverse word frequency similarity model, determine the part-of-speech word frequency inverse word frequency similarity between the target text and the candidate text.

[0090] The model for determining the similarity of word frequency inverse frequency with part of speech in this embodiment also samples the word frequency inverse frequency model TF-IDF model, but the model parameter ∑ k n k,j and |{j:t i ∈d j}|will increase due to the introduction of part of speech. Then, the process of determining the similarity of the inverse frequency of part-of-speech words is as follows: input each target part-of-speech tagged word, the target text and the text database into the TF-IDF model, obtain the TF-IDF value of each target part-of-speech tagged word to form the inverse frequency feature vector of the target text, the number of columns of the vector is consistent with the vector dimension determined above, wherein the TF-IDF value is filled in the element position corresponding to the target part-of-speech word, and the remaining element positions are filled with 0. Similarly, the inverse frequency feature vector of the part-of-speech word corresponding to the alternative text can be obtained by performing word segmentation, part-of-speech tagging and part-of-speech tagging on the alternative text. Finally, the vector cosine of the inverse frequency feature vector of the part-of-speech word of the target text and the inverse frequency feature vector of the part-of-speech word corresponding to the alternative text is calculated to obtain the inverse frequency similarity of the part-of-speech word of the target text and the alternative text.

[0091] Exemplarily, S340 includes: inputting each target part-of-speech tagged word and the target text into a word frequency inverse word frequency similarity model to generate a part-of-speech word frequency inverse word frequency feature vector corresponding to the target text; determining the part-of-speech word frequency inverse word frequency similarity between the target text and the alternative text corresponding to the row vector based on the part-of-speech word frequency inverse word frequency feature vector and the row vector in the part-of-speech word frequency inverse word frequency feature matrix; wherein the part-of-speech word frequency inverse word frequency feature matrix is ​​generated based on the part-of-speech word frequency segmentation results of the text database and the word frequency inverse word frequency similarity model.

[0092] When each text in the text database is an alternative text, in order to improve the operation efficiency, each alternative text in the text database can be pre-processed by word segmentation, part-of-speech tagging and data cleaning to obtain the corresponding part-of-speech word segmentation results. Then input it into the TF-IDF model to obtain the part-of-speech word frequency inverse word frequency feature matrix of the text database. The number of columns of the part-of-speech word frequency inverse word frequency feature matrix is ​​consistent with the number of non-repeating words without repeated parts of speech in the text database. The element position without corresponding part-of-speech words is filled with 0, and each row vector in the matrix represents the feature vector of an alternative text. Finally, the vector cosine between the part-of-speech word frequency inverse word frequency feature vector and each row vector in the part-of-speech word frequency inverse word frequency feature matrix is ​​calculated respectively, and the part-of-speech word frequency inverse word frequency similarity between the target text and each alternative text can be obtained.

[0093] S350, determining the text similarity between the target text and the candidate texts based on the preset word semantic weight, the preset part-of-speech word frequency inverse word frequency weight, the word semantic similarity and the part-of-speech word frequency inverse word frequency similarity.

[0094] The preset word frequency inverse word frequency weight and word frequency inverse word frequency similarity in formula (1) are respectively determined as the preset part-of-speech word frequency inverse word frequency weight w 22 Similarity score with the inverse frequency of part-of-speech words 22 , the text similarity between the target text and the candidate text can be obtained according to the following formula:

[0095] score=w 1 score 1 +w 22 score 22 (8)

[0096] The technical solution of this embodiment determines the word semantic similarity and the inverse frequency similarity of part-of-speech words between the target text and the alternative texts, and determines the text similarity between the target text and the alternative texts based on preset word semantic weights, preset inverse frequency weights of part-of-speech words, and word semantic similarity and inverse frequency similarity of part-of-speech words, thereby realizing the determination of the text similarity of the target text from two dimensions of word semantics and part-of-speech word importance, and improving the accuracy of determining text similarity to a certain extent.

[0097] Embodiment 4

[0098] Based on the above-mentioned embodiment 1, this embodiment further optimizes "determining the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text", and optimizes "determining the text similarity between the target text and the candidate text according to the preset word semantic weight, the preset word frequency inverse word frequency weight, and the word semantic similarity and word frequency inverse word frequency similarity". The explanations of the terms that are the same or corresponding to the above-mentioned embodiments are not repeated here. Figure 6a , the text similarity determination method provided in this embodiment includes:

[0099] S410: Obtain the target text and the candidate texts whose similarities are to be determined.

[0100] S420: Segment and tag the target text to obtain target part-of-speech words and target part-of-speech tagged words corresponding to the target text.

[0101] See also Figure 6b ,The target text is processed by word segmentation to obtain each target part-of-speech-free word, and after word segmentation and part-of-speech tagging, each target part-of-speech tagged word is obtained.

[0102] S430 , determining the word semantic similarity between the target text and the candidate text based on each target part-of-speech word and the candidate text and based on a pre-trained word semantic similarity model.

[0103] The word semantic feature vector of the target text can be obtained from each target word without part of speech and the trained Word2vec model. The word semantic similarity between the word semantic feature vector and the row vector in the word semantic feature vector corresponding to the text database is calculated to obtain the word semantic similarity score between the target text and the candidate text in the text database. 1 If there are n candidate texts in the text database, then n scores can be obtained. 1 .

[0104] S440, according to each target part-of-speech word, the target text and the candidate text, based on the word frequency inverse word frequency similarity model, determine the part-of-speech word frequency inverse word frequency similarity between the target text and the candidate text.

[0105] The target part-of-speech word and the trained TF-IDF model can be used to obtain the inverse frequency feature vector of the target text. The similarity between the inverse frequency feature vector of the target text and the row vector of the inverse frequency feature vector of the target text can be calculated to obtain the similarity score of the target text and the candidate text in the text database. 21 If there are n candidate texts in the text database, then n scores can be obtained. 21 .

[0106] S450, according to each target part-of-speech tagged word, the target text and the candidate text, based on the word frequency inverse word frequency similarity model, determine the part-of-speech word frequency inverse word frequency similarity between the target text and the candidate text.

[0107] The target text's part-of-speech word frequency inverse word frequency feature vector can be obtained from each target part-of-speech word tag and the trained TF-IDF model. The similarity between the target text and the candidate text in the text database can be obtained by calculating the part-of-speech word frequency inverse word frequency similarity score. 22 If there are n candidate texts in the text database, then n scores can be obtained. 22 .

[0108] S460, determining the text similarity between the target text and the candidate texts based on the preset word semantic weight, the preset inverse word frequency weight of the word without part of speech, the preset inverse word frequency weight of the word with part of speech, and the word semantic similarity, the inverse word frequency similarity of the word without part of speech and the inverse word frequency similarity of the word with part of speech.

[0109] The preset word frequency inverse word frequency weight in formula (1) is determined as the preset word frequency inverse word frequency weight w 21 and the preset part-of-speech word frequency inverse word frequency weight w 22 , the word frequency inverse word frequency similarity is determined as the word frequency inverse word frequency similarity score without part of speech 21 Similarity score with the inverse frequency of part-of-speech words 22 , the text similarity between the target text and the candidate text can be obtained according to the following formula:

[0110] score=w 1 score 1 +w 21 score 21 +w 22 score 22 (9)

[0111] If there are n candidate texts in the text database, then n scores can be obtained through formula (9), and each score represents the text similarity between the target text and the corresponding candidate text.

[0112] S470: Arrange the candidate texts in descending order according to the similarities of the texts to generate a sorting result.

[0113] After determining the similarity of each text, multiple candidate texts whose similarity meets the business requirements can be determined from all candidate texts. At this time, in order to further improve the business operation efficiency, all candidate texts can be sorted in descending order according to text similarity to generate a sorting result.

[0114] S480: extracting a preset number of candidate texts from the sorting results as similar texts to the target text.

[0115] The number of candidate texts to be selected is determined according to business requirements, that is, the preset number. Then, the preset number of candidate texts ranked top are extracted from the sorting results as similar texts that are relatively similar to the target text.

[0116] S490: When the business scenario is an intelligent question-answering scenario and the business requirement is to determine alternative answers for a target text, an answer corresponding to each similar text is extracted from a short text database as an alternative answer for the target text.

[0117] In the business scenario of intelligent question and answer, since questions and answers are usually short texts, the target text is the target short text. If the business requirement is only to determine similar short texts, then S480 can end. However, if the business requirement is to determine the alternative answers to the target text, then the alternative text needs to be a short text in the short text database. At this time, it is necessary to extract the answer short text for each similar text from the short text database as an alternative answer to the target text, so as to provide users with more accurate answers in a faster and more convenient way in intelligent customer service, or to assist manual customer service in providing answers to similar questions to manual customer service.

[0118] The technical solution of this embodiment determines the text similarity between the target text and the candidate text by determining the word semantic similarity, the inverse word frequency similarity of the word frequency without part of speech and the inverse word frequency similarity of the word frequency with part of speech, and determines the text similarity between the target text and the candidate text according to the preset word semantic weight, the preset inverse word frequency weight of the word frequency without part of speech, the preset inverse word frequency weight of the word frequency with part of speech, and the word semantic similarity, the inverse word frequency similarity of the word frequency without part of speech and the inverse word frequency similarity of the word frequency with part of speech, thereby realizing the determination of the text similarity of the target text from three dimensions of word semantics, importance of the word without part of speech and importance of the word with part of speech, and improving the determination accuracy of the text similarity to a greater extent. By sorting the candidate texts according to the text similarity and determining the answer of the similar text with the top ranking as the candidate answer of the target text, the determination accuracy and efficiency of the candidate answer in the intelligent question and answer system can be improved.

[0119] Embodiment 5

[0120] This embodiment provides a text similarity determination device. Figure 7 , the device specifically comprises:

[0121] A target text determination module 710 is used to obtain a target text and an alternative text whose similarity is to be determined;

[0122] The first similarity determination module 720 is used to determine the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text, wherein the word frequency inverse word frequency similarity includes word frequency inverse word frequency similarity without part of speech and / or word frequency inverse word frequency similarity with part of speech;

[0123] The second similarity determination module 730 is used to determine the text similarity between the target text and the alternative text based on the preset word semantic weight, the preset word frequency inverse word frequency weight, the word semantic similarity and the word frequency inverse word frequency similarity, wherein the preset word frequency inverse word frequency weight includes the preset part-of-speech word frequency inverse word frequency weight and / or the preset part-of-speech word frequency inverse word frequency weight.

[0124] Optionally, the first similarity determination module 720 is specifically configured to:

[0125] Segment the target text to obtain the target part-of-speech words corresponding to the target text;

[0126] According to each target part-of-speech word and the candidate text, based on the pre-trained word semantic similarity model, determine the word semantic similarity between the target text and the candidate text;

[0127] According to each target part-of-speech word, the target text and the candidate text, based on the word frequency inverse word frequency similarity model, determine the part-of-speech word frequency inverse word frequency similarity between the target text and the candidate text;

[0128] Accordingly, the second similarity determination module 730 is specifically configured to:

[0129] The text similarity between the target text and the candidate text is determined based on the preset word semantic weight, the preset inverse word frequency weight of the word without part of speech, the word semantic similarity and the inverse word frequency similarity of the word without part of speech.

[0130] Optionally, the first similarity determination module 720 is specifically configured to:

[0131] Perform word segmentation and part-of-speech tagging on the target text to obtain each target part-of-speech-free word and each target part-of-speech tagged word corresponding to the target text;

[0132] According to each target part-of-speech word and the candidate text, based on the pre-trained word semantic similarity model, determine the word semantic similarity between the target text and the candidate text;

[0133] According to each target part-of-speech tagged word, the target text and the candidate text, based on the word frequency inverse word frequency similarity model, determine the word frequency inverse word frequency similarity between the target text and the candidate text;

[0134] Accordingly, the second similarity determination module 730 is specifically configured to:

[0135] The text similarity between the target text and the candidate text is determined based on the preset word semantic weight, the preset part-of-speech word frequency inverse word frequency weight, the word semantic similarity and the part-of-speech word frequency inverse word frequency similarity.

[0136] Optionally, the first similarity determination module 720 is specifically configured to:

[0137] Perform word segmentation and part-of-speech tagging on the target text to obtain each target part-of-speech-free word and each target part-of-speech tagged word corresponding to the target text;

[0138] According to each target part-of-speech word and the candidate text, based on the pre-trained word semantic similarity model, determine the word semantic similarity between the target text and the candidate text;

[0139] According to each target part-of-speech word, the target text and the candidate text, based on the word frequency inverse word frequency similarity model, determine the part-of-speech word frequency inverse word frequency similarity between the target text and the candidate text;

[0140] According to each target part-of-speech tagged word, the target text and the candidate text, based on the word frequency inverse word frequency similarity model, determine the word frequency inverse word frequency similarity between the target text and the candidate text;

[0141] Accordingly, the second similarity determination module 730 is specifically configured to:

[0142] The text similarity between the target text and the alternative texts is determined based on the preset word semantic weight, the preset inverse word frequency weight of the word without part of speech, the preset inverse word frequency weight of the word with part of speech, as well as the word semantic similarity, the inverse word frequency similarity of the word without part of speech and the inverse word frequency similarity of the word with part of speech.

[0143] Optionally, the word frequency-inverse word frequency similarity model is a word frequency-inverse word frequency model, and the word semantic similarity model is a Word2vec model;

[0144] The Word2vec model is pre-trained based on a short text database and a long text database, wherein the long text database is pre-built based on business data corresponding to business scenarios, and the short text database is pre-built based on business data corresponding to business needs under business scenarios.

[0145] Optionally, based on the above device, the device further includes a similar text determination module, which is used to:

[0146] When there are multiple candidate texts, after determining the text similarity between the target text and the candidate texts, the candidate texts are sorted in descending order according to their text similarities to generate a sorting result;

[0147] A preset number of alternative texts are extracted from the sorting results as similar texts to the target text.

[0148] Furthermore, based on the above device, the device also includes an alternative answer determination module, which is used to:

[0149] When the business scenario is an intelligent question-and-answer scenario and the business requirement is to determine alternative answers for a target text, the target text is a target short text, and the alternative texts are short texts in a short text database. After extracting a preset number of alternative texts from the sorting results as similar texts to the target text, the answer corresponding to each similar text is extracted from the short text database as an alternative answer to the target text.

[0150] Through a text similarity determination device of Example 5 of the present invention, it is possible to grasp the overall semantics of sentences in a text by utilizing multiple dimensional features of word semantics, word frequency and / or part of speech, thereby characterizing text similarity from different dimensions, obtaining the comprehensive similarity of the text, and greatly improving the accuracy of text similarity.

[0151] The text similarity determination device provided in the embodiment of the present invention can execute the text similarity determination method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0152] It is worth noting that in the embodiment of the above-mentioned text similarity determination device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0153] Embodiment 6

[0154] See also Figure 8 This embodiment provides a device, which includes: one or more processors 820; a storage device 810, which is used to store one or more programs. When the one or more programs are executed by the one or more processors 820, the one or more processors 820 implement the text similarity determination method provided by the embodiment of the present invention, including:

[0155] Obtaining a target text and an alternative text whose similarity is to be determined;

[0156] Determine the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text, wherein the word frequency inverse word frequency similarity includes word frequency inverse word frequency similarity without part of speech and / or word frequency inverse word frequency similarity with part of speech;

[0157] The text similarity between the target text and the alternative text is determined based on the preset word semantic weight, the preset word frequency inverse word frequency weight, the word semantic similarity and the word frequency inverse word frequency similarity, wherein the preset word frequency inverse word frequency weight includes the preset word frequency inverse word frequency weight without part of speech and / or the preset word frequency inverse word frequency weight with part of speech.

[0158] Of course, those skilled in the art can understand that the processor 820 can also implement the technical solution of the text similarity determination method provided by any embodiment of the present invention.

[0159] Figure 8 The device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention. Figure 8 As shown, the device includes a processor 820, a storage device 810, an input device 830 and an output device 840; the number of processors 820 in the device can be one or more. Figure 8 A processor 820 is taken as an example; the processor 820, the storage device 810, the input device 830 and the output device 840 in the device can be connected via a bus or other means. Figure 8 The connection via bus 850 is taken as an example.

[0160] The storage device 810, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the text similarity determination method in the embodiment of the present invention (for example, the target text acquisition module, the first similarity determination module and the second similarity determination module in the text similarity determination device).

[0161] The storage device 810 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the storage device 810 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 810 may further include a memory remotely arranged relative to the processor 820, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The input device 830 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 840 may include a display device such as a display screen.

[0163] Embodiment 7

[0164] This embodiment provides a storage medium containing computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to perform a text similarity determination method. The method includes:

[0165] Obtaining a target text and an alternative text whose similarity is to be determined;

[0166] Determine the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text, wherein the word frequency inverse word frequency similarity includes word frequency inverse word frequency similarity without part of speech and / or word frequency inverse word frequency similarity with part of speech;

[0167] The text similarity between the target text and the alternative text is determined based on the preset word semantic weight, the preset word frequency inverse word frequency weight, the word semantic similarity and the word frequency inverse word frequency similarity, wherein the preset word frequency inverse word frequency weight includes the preset word frequency inverse word frequency weight without part of speech and / or the preset word frequency inverse word frequency weight with part of speech.

[0168] Of course, the computer executable instructions of a storage medium including computer executable instructions provided in an embodiment of the present invention are not limited to the above method operations, and can also execute related operations in the text similarity determination method provided in any embodiment of the present invention.

[0169] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present invention can be implemented by software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a device (which can be a personal computer, server, or network device, etc.) to perform the text similarity determination method provided by each embodiment of the present invention.

[0170] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for determining text similarity, characterized in that: include: Obtaining a target text and an alternative text whose similarity is to be determined; Determine the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text, wherein the word frequency inverse word frequency similarity includes word frequency inverse word frequency similarity without part of speech and / or word frequency inverse word frequency similarity with part of speech; The text similarity between the target text and the alternative text is determined based on a preset word semantic weight, a preset word frequency inverse word frequency weight, and the word semantic similarity and the word frequency inverse word frequency similarity, wherein the preset word frequency inverse word frequency weight includes a preset part-of-speech word frequency inverse word frequency weight and / or a preset part-of-speech word frequency inverse word frequency weight; the preset word semantic weight and the preset word frequency inverse word frequency weight are respectively used to determine the proportion of the word semantic similarity and the word frequency inverse word frequency similarity in the process of determining the text similarity.

2. The method according to claim 1, characterized in that Determining the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text includes: Segmenting the target text to obtain target part-of-speech words corresponding to the target text; According to each of the target part-of-speech words and the candidate texts, based on a pre-trained word semantic similarity model, determine the word semantic similarity between the target text and the candidate text; According to each of the target part-of-speech words, the target text and the candidate texts, based on a word frequency inverse word frequency similarity model, determine the inverse word frequency similarity of the part-of-speech words between the target text and the candidate texts; Accordingly, determining the text similarity between the target text and the candidate text according to the preset word semantic weight, the preset word frequency inverse word frequency weight, the word semantic similarity and the word frequency inverse word frequency similarity includes: The text similarity between the target text and the candidate text is determined based on the preset word semantic weight, the preset inverse word frequency weight of the word without part of speech, the word semantic similarity and the inverse word frequency similarity of the word without part of speech.

3. The method according to claim 1, characterized in that Determining the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text includes: Performing word segmentation and part-of-speech tagging on the target text to obtain target part-of-speech words and target part-of-speech tagged words corresponding to the target text; According to each of the target part-of-speech words and the candidate texts, based on a pre-trained word semantic similarity model, determine the word semantic similarity between the target text and the candidate text; According to each of the target part-of-speech tagged words, the target text and the candidate text, based on a word frequency inverse word frequency similarity model, determine the word frequency inverse word frequency similarity between the target text and the candidate text; Accordingly, determining the text similarity between the target text and the candidate text according to the preset word semantic weight, the preset word frequency inverse word frequency weight, the word semantic similarity and the word frequency inverse word frequency similarity includes: The text similarity between the target text and the candidate text is determined based on the preset word semantic weight, the preset part-of-speech word frequency inverse word frequency weight, the word semantic similarity and the part-of-speech word frequency inverse word frequency similarity.

4. The method according to claim 1, characterized in that: Determining the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text includes: Performing word segmentation and part-of-speech tagging on the target text to obtain target part-of-speech words and target part-of-speech tagged words corresponding to the target text; According to each of the target part-of-speech words and the candidate texts, based on a pre-trained word semantic similarity model, determine the word semantic similarity between the target text and the candidate text; According to each of the target part-of-speech words, the target text and the candidate texts, based on a word frequency inverse word frequency similarity model, determine the inverse word frequency similarity of the part-of-speech words between the target text and the candidate texts; According to each of the target part-of-speech tagged words, the target text and the candidate text, based on a word frequency inverse word frequency similarity model, determine the word frequency inverse word frequency similarity between the target text and the candidate text; Accordingly, determining the text similarity between the target text and the candidate text according to the preset word semantic weight, the preset word frequency inverse word frequency weight, the word semantic similarity and the word frequency inverse word frequency similarity includes: The text similarity between the target text and the candidate text is determined based on the preset word semantic weight, the preset inverse word frequency weight of the word without part of speech, the preset inverse word frequency weight of the word with part of speech, and the word semantic similarity, the inverse word frequency similarity of the word without part of speech and the inverse word frequency similarity of the word with part of speech.

5. The method according to any one of claims 2 to 4, characterized in that: The word frequency inverse word frequency similarity model is a word frequency inverse word frequency model, and the word semantic similarity model is a Word2vec model; The Word2vec model is pre-trained based on a short text database and a long text database, wherein the long text database is pre-constructed based on business data corresponding to a business scenario, and the short text database is pre-constructed based on business data corresponding to business needs in the business scenario.

6. The method according to claim 1, characterized in that When there are multiple candidate texts, after determining the text similarity between the target text and the candidate texts, the method further includes: Arrange the candidate texts in descending order according to the similarities of the texts to generate a sorting result; A preset number of the candidate texts are extracted from the sorting results as similar texts to the target text.

7. The method according to claim 6, characterized in that When the business scenario is an intelligent question-answering scenario and the business requirement is to determine alternative answers to the target text, the target text is a target short text, and the alternative texts are short texts in a short text database, after extracting a preset number of the alternative texts from the sorting results as similar texts to the target text, the method further includes: The answer corresponding to each of the similar texts is extracted from the short text database as an alternative answer to the target text.

8. A device for determining text similarity, characterized in that: include: A target text acquisition module, used to acquire a target text and an alternative text whose similarity is to be determined; A first similarity determination module is used to determine the word semantic similarity and word frequency inverse word frequency similarity between the target text and the candidate text, wherein the word frequency inverse word frequency similarity includes word frequency inverse word frequency similarity without part of speech and / or word frequency inverse word frequency similarity with part of speech; The second similarity determination module is used to determine the text similarity between the target text and the alternative text based on a preset word semantic weight, a preset word frequency inverse word frequency weight, and the word semantic similarity and the word frequency inverse word frequency similarity, wherein the preset word frequency inverse word frequency weight includes a preset part-of-speech word frequency inverse word frequency weight and / or a preset part-of-speech word frequency inverse word frequency weight; the preset word semantic weight and the preset word frequency inverse word frequency weight are respectively used to determine the proportion of the word semantic similarity and the word frequency inverse word frequency similarity in the process of determining the text similarity.

9. A device, characterized in that: The device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the text similarity determination method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for determining text similarity as described in any one of claims 1 to 7 is implemented.

Images