Text similarity determination method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a text similarity determination method and device, equipment, a storage medium and a program product, and relates to the field of big data. The method comprises the following steps: performing abstract extraction processing on a first text and a second text to obtain a first abstract of the first text and a second abstract of the second text; performing text fingerprint extraction processing on the first text, the second text, the first abstract and the second abstract to obtain a first fingerprint of the first text, a second fingerprint of the second text, a third fingerprint of the first abstract and a fourth fingerprint of the second abstract; determining a first similarity according to the first fingerprint and the second fingerprint, and determining a second similarity according to the third fingerprint and the fourth fingerprint; and determining the text similarity between the first text and the second text according to the first similarity and the second similarity. According to the method provided by the embodiment of the invention, the accuracy of text similarity calculation is improved.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a method, device, equipment, storage medium and program product for determining text similarity. Background Art

[0002] As a key technology to measure the relevance between the content or semantics of two texts, text similarity calculation is widely used in natural language processing (NLP), data retrieval and data mining.

[0003] In the related art, the text similarity between two texts can be determined by constructing a similarity calculation method or training a similarity calculation model.

[0004] However, the similarity calculation method or similarity calculation model has low accuracy when calculating text similarity. Summary of the invention

[0005] The present application provides a text similarity determination method, apparatus, device, storage medium and program product to solve the technical problem that similarity calculation methods or similarity calculation models in related technologies have low accuracy when calculating text similarity.

[0006] In a first aspect, the present application provides a method for determining text similarity, the method comprising:

[0007] Performing summary extraction processing on the first text and the second text to obtain a first summary of the first text and a second summary of the second text;

[0008] Performing text fingerprint extraction processing on the first text, the second text, the first summary and the second summary respectively to obtain a first fingerprint of the first text, a second fingerprint of the second text, a third fingerprint of the first summary and a fourth fingerprint of the second summary;

[0009] Determine a first similarity based on the first fingerprint and the second fingerprint, and determine a second similarity based on the third fingerprint and the fourth fingerprint;

[0010] The text similarity between the first text and the second text is determined according to the first similarity and the second similarity.

[0011] In a second aspect, the present application provides a text similarity determination device, comprising:

[0012] An extraction module, used for performing summary extraction processing on the first text and the second text to obtain a first summary of the first text and a second summary of the second text;

[0013] The extraction module is further used to perform text fingerprint extraction processing on the first text, the second text, the first summary and the second summary respectively to obtain a first fingerprint of the first text, a second fingerprint of the second text, a third fingerprint of the first summary and a fourth fingerprint of the second summary;

[0014] a determination module, configured to determine a first similarity based on the first fingerprint and the second fingerprint, and to determine a second similarity based on the third fingerprint and the fourth fingerprint;

[0015] The determination module is further used to determine the text similarity between the first text and the second text according to the first similarity and the second similarity.

[0016] In a third aspect, an embodiment of the present application provides a device for determining text similarity, including: a processor, and a memory in communication with the processor;

[0017] Memory stores computer-executable instructions;

[0018] The processor executes the computer-executable instructions stored in the memory to implement the text similarity determination method involved in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the text similarity determination method involved in the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the text similarity determination method involved in the first aspect.

[0021] The text similarity determination method, apparatus, device, storage medium and program product provided by the present application can extract key semantic information in a first text as a first summary, and extract key semantic information of a second text as a second summary, and perform text fingerprint extraction processing on the first text, the second text, the first summary and the second summary respectively to obtain a first fingerprint of the first text, a second fingerprint of the second text, a third fingerprint of the first summary and a fourth fingerprint of the second summary; determine a first similarity based on the first fingerprint and the second fingerprint, and determine a second similarity based on the third fingerprint and the fourth fingerprint; and comprehensively determine the text similarity between the first text and the second text in combination with the first similarity and the second similarity, in which the similarity analysis of the semantic information of the two texts is integrated to achieve a comprehensive comparison of the text content and semantic information of the two texts, which is conducive to improving the accuracy of the text similarity calculation of the two texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0023] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0024] Figure 2 One of the flow charts of the method for determining text similarity provided in the embodiment of the present application;

[0025] Figure 3 A second flowchart of a method for determining text similarity provided in an embodiment of the present application;

[0026] Figure 4 A third flowchart of a method for determining text similarity provided in an embodiment of the present application;

[0027] Figure 5 A fourth flowchart of a method for determining text similarity provided in an embodiment of the present application;

[0028] Figure 6 A schematic diagram of the structure of the similarity determination device provided in this application;

[0029] Figure 7 A schematic diagram of the structure of a text similarity determination device provided in this application.

[0030] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0033] In addition, this application involves big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and the use of artificial intelligence technology for automated decision-making, and a technical solution for making decisions that have a significant impact on personal rights and interests based on the results of automated decision-making. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decision-making; if the user chooses to reject, the expert decision-making process will be entered.

[0034] It should be noted that the text similarity determination method, device, equipment, storage medium and program product provided in the present application can be used in the field of big data, and can also be used in any field other than big data. The application does not limit the application field of the text similarity determination method, device, equipment, storage medium and program product.

[0035] For ease of understanding, the following Figure 1 , briefly describe the application scenarios to which the embodiments of the present application are applicable.

[0036] Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application, such as Figure 1 As shown, in the text similarity determination device, a similarity comparison process is performed on the first text and the second text to obtain the text similarity between the first text and the second text.

[0037] In this scenario, the text similarity determination device can use the text similarity determination method as shown below to determine the text similarity between two texts: split each text into multiple words, and construct a word vector corresponding to each text based on the multiple words, so as to determine the text similarity between the two texts through the word vector corresponding to the first text and the word vector corresponding to the second text.

[0038] In combination with the above scenarios, it can be seen that the text similarity determination method in the related art usually determines the text similarity between two texts based on the similarity between the words of the two texts. However, in some scenarios, the semantics represented by the same word may be different in different contexts, resulting in low accuracy when determining the text similarity between two texts based on the text similarity determination method.

[0039] The text similarity determination method provided in the embodiment of the present application can extract key semantic information in the first text as a first summary, and extract key semantic information in the second text as a second summary, and calculate the first similarity of the two texts and the second similarity of the two summaries, and comprehensively determine the text similarity between the first text and the second text by combining the first similarity and the second similarity. In this process, similarity analysis of the semantic information of the two texts is integrated to achieve a comprehensive comparison of the text content and semantic information of the two texts, which is conducive to improving the accuracy of text similarity calculation of the two texts.

[0040] It should be noted that Figure 1 It is only a schematic diagram of a scenario in the form of an example, and is not intended to limit the scenario schematic diagram provided in the embodiments of the present application.

[0041] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0042] Figure 2 One of the flow charts of the method for determining text similarity provided in the embodiment of the present application is as follows: Figure 2 As shown, the method includes:

[0043] S201 , performing summary extraction processing on a first text and a second text to obtain a first summary of the first text and a second summary of the second text.

[0044] The text similarity determination device may provide a data interface externally, through which the text similarity determination device may receive a first text and a second text sent or uploaded by a user, wherein the first text or the second text may be a piece of text data. For example, the first text may be an article.

[0045] In some embodiments, a summary extraction model may be used to perform summary extraction processing on the first text and the second text. The summary extraction model may be used to extract key semantic information in the text and organize the extracted key semantic information into a summary corresponding to the text.

[0046] It can be understood that when using the summary extraction model to perform summary extraction processing on a text, it is possible to first detect whether summary information already exists in the text. For example, a paper usually includes summary information. When it is determined that summary information already exists in the text, the summary information can be directly extracted. When it is determined that no summary information exists in the text, it is necessary to perform semantic analysis and key information extraction on the text to obtain the summary information of the text.

[0047] In some embodiments, a summary extraction model that has been trained in the relevant technology can be used; or, a plurality of text data can be obtained in a text library to generate training data, and the initial deep learning model is trained based on the training data and a plurality of semantic recognition and extraction algorithms to obtain the summary extraction model. Exemplarily, the initial deep learning model may include at least one of the following: a convolutional neural network (CNN), a recurrent neural network (RNN), or a neural network model of a self-attention mechanism.

[0048] It is understandable that when the number of training data samples is insufficient, the computational accuracy of the trained summary extraction model will be low. In view of this, the method can increase the amount of training data when training the summary extraction model to ensure the computational accuracy of the summary extraction model.

[0049] It should be noted that in some examples, terms such as "summary extraction model" and "semantic extraction model" and "information extraction model" can be used interchangeably.

[0050] S202: Perform text fingerprint extraction processing on the first text, the second text, the first summary, and the second summary, respectively, to obtain a first fingerprint of the first text, a second fingerprint of the second text, a third fingerprint of the first summary, and a fourth fingerprint of the second summary.

[0051] The first fingerprint, the second fingerprint, the third fingerprint and the fourth fingerprint may be represented by a vector matrix, and the vector matrix may include multiple elements. For example, the first fingerprint may include 5 elements, and the first fingerprint may be represented as [a1a2 a3 a4 a5].

[0052] Optionally, the method can perform text fingerprint extraction processing on the first text, the second text, the first summary and the second summary in parallel to improve the efficiency of the text fingerprint extraction processing; the method can also perform text fingerprint extraction processing on the first text, the second text, the first summary and the second summary in sequence to reduce resource consumption in the text fingerprint extraction processing.

[0053] It should be noted that the process of extracting text fingerprints from the first text, the second text, the first summary, and the second summary is the same. Figure 3 In the text fingerprint extraction process of the first text, the process of determining the first fingerprint of the first text will be described in detail by taking the text fingerprint extraction process of the first text as an example. The process of performing text fingerprint extraction process on the second text, the first summary and the second summary can refer to the process of performing text fingerprint extraction process on the first text, and will not be described in detail.

[0054] S203: Determine a first similarity according to the first fingerprint and the second fingerprint, and determine a second similarity according to the third fingerprint and the fourth fingerprint.

[0055] Optionally, a similarity calculation method may be used to determine the first similarity and the second similarity. Exemplarily, the similarity calculation method may include at least one of the following: a cosine similarity method, an edit distance calculation method, a Hamming distance method, a hash algorithm for estimating set similarity, a hash algorithm for duplicate detection, or a longest sentence signature algorithm.

[0056] For example, the similarity calculation method is a Hamming distance method, and a first Hamming distance can be determined according to the first fingerprint and the second fingerprint based on the Hamming distance method, and a second Hamming distance can be determined according to the first fingerprint and the second fingerprint based on the Hamming distance method.

[0057] A similarity calculation model may be used to determine the first similarity and the second similarity. For example, the similarity calculation model may be a model based on an attention mechanism and a convolutional neural network.

[0058] S204: Determine text similarity between the first text and the second text according to the first similarity and the second similarity.

[0059] In some embodiments, a text weight and a summary weight may be determined; the first similarity may be weighted according to the text weight to obtain a first weighted similarity; the second similarity may be weighted according to the summary weight to obtain a second weighted similarity; and the text similarity may be obtained based on the first weighted similarity and the second weighted similarity.

[0060] The user can pre-configure the text weight and summary weight according to the requirements, so as to adjust the influence of the text content on the text similarity based on the text weight, and adjust the influence of the text semantic information on the text similarity based on the summary weight.

[0061] The text weight and summary weight may satisfy the following relationship: text weight + summary weight = preset value, for example, the preset value may be 1 or 100%. The text weight and summary weight may be the same, for example, both the text weight and summary weight are 0.5 (or 50%). The text weight and summary weight may also be different, the text weight may be greater than the summary weight, for example, the text weight is 0.6 (or 60%), and the summary weight is 0.4 (or 40%); the text weight may also be less than the summary weight, for example, the text weight is 0.4 (or 40%), and the summary weight is 0.6 (or 60%).

[0062] In some embodiments, the text similarity may be the sum of the first weighted similarity and the second weighted similarity, that is, text similarity=first weighted similarity+second weighted similarity.

[0063] In the process of calculating the text similarity between the first text and the second text, the method comprehensively considers the influence of the content of the text and the semantic information of the text on the text similarity through the text weight and the summary weight, which is conducive to improving the accuracy of text similarity calculation.

[0064] The text similarity determination method provided in this embodiment can extract key semantic information in the first text as a first summary, and extract key semantic information in the second text as a second summary, and calculate the first similarity of the two texts and the second similarity of the two summaries, and comprehensively determine the text similarity between the first text and the second text by combining the first similarity and the second similarity. In this process, similarity analysis of the semantic information of the two texts is integrated to achieve a comprehensive comparison of the text content and semantic information of the two texts, which is conducive to improving the accuracy of text similarity calculation of the two texts.

[0065] Based on the above embodiments, the text similarity determination method provided in the embodiments of the present application also involves a process of performing text fingerprint extraction processing on the text or abstract. Figure 3 , taking the process of extracting the text fingerprint of the first text as an example, the process of extracting the text fingerprint of the text or summary is described in detail. The process of extracting the text fingerprint of the second text, the first summary and the second summary can refer to Figure 3 The process of extracting and processing the text fingerprint of the first text in the embodiment will not be described in detail later.

[0066] Figure 3 The second flowchart of the method for determining text similarity provided in the embodiment of the present application is as follows: Figure 3 As shown, the embodiment of the present application is Figure 2 Based on the embodiment, a method for determining text similarity is described in detail, and the method includes:

[0067] S301 : Preprocess a first text to obtain a plurality of keywords of the first text, and a sentence component identifier and a part-of-speech identifier of each keyword.

[0068] The sentence component identifier may include at least one of the following: subject, predicate, object, attributive, adverbial or complement; the part-of-speech identifier may include at least one of the following: noun, pronoun, verb, adjective, adverb, preposition, conjunction, interjection, numeral, quantifier or stop word.

[0069] In some embodiments, the first text can be segmented to obtain multiple candidate words; sentence component and part-of-speech tagging can be performed on each of the multiple candidate words to obtain a sentence component identifier and a part-of-speech identifier corresponding to each candidate word; and the multiple candidate words can be filtered based on the sentence component identifier and part-of-speech identifier corresponding to each candidate word to obtain multiple keywords, as well as a sentence component identifier and a part-of-speech identifier for each keyword.

[0070] Optionally, some word segmentation models for word segmentation processing may be used to perform word segmentation processing on the first text, for example, the word segmentation model may be a Word2vec model; some tools for sentence component analysis and part-of-speech tagging may be used to perform sentence component and part-of-speech tagging processing on each candidate word.

[0071] Optionally, a word segmentation algorithm in the related art may be used to perform word segmentation processing on the first text, and the word segmentation algorithm may be used to divide the text into substrings of fixed length.

[0072] It is understandable that when calculating text similarity, the candidate words corresponding to some sentence component identifiers and part-of-speech identifiers have a lower impact on text similarity. These sentence component identifiers can be determined as sentence component identifiers to be filtered, and these part-of-speech identifiers can be determined as part-of-speech identifiers to be filtered. Based on the sentence component identifiers to be filtered and the part-of-speech identifiers to be filtered, multiple candidate words can be filtered.

[0073] Exemplarily, the part-of-speech identifiers to be filtered may include prepositions, interjections, adverbs, and stop words; the sentence component identifiers to be filtered may include attributives, adverbials, and complements.

[0074] The method sequentially performs word segmentation processing, sentence component and part-of-speech tagging processing, and filtering processing on the first text, which is conducive to determining multiple keywords that have an important influence on calculating text similarity among multiple candidate words of the first text, so as to clarify the synonyms, antonyms, subject, predicate, object, semantic context and other relationships in the text in the similarity calculation, thereby improving the accuracy of text similarity calculation; by filtering other candidate words with lower influence on calculating text similarity, the amount of data in subsequent text similarity comparison calculation is reduced, thereby improving the efficiency of text similarity calculation.

[0075] S302: Determine a plurality of word weights corresponding to the plurality of keywords according to the sentence component identifiers and part-of-speech identifiers of the plurality of keywords.

[0076] In some embodiments, different sentence components and parts of speech have different effects on calculating text similarity. When different weighting algorithms are used to process the same keyword, the weight of the keyword obtained is also different. This method can comprehensively determine the word weight corresponding to each keyword by combining the sentence component identifier and part of speech identifier of each keyword and multiple weighting algorithms to further improve the accuracy of determining the word weight corresponding to each keyword.

[0077] Optionally, an initial word vector of the first text and a vector space of word weights corresponding to the initial word vector may be constructed according to multiple keywords and multiple word weights corresponding to the multiple keywords. For example, the vector space may be expressed as:

[0078]

[0079] Among them, k1, k2 and k i Respectively represent the initial word vectors corresponding to the 1st, 2nd and i-th keywords of the text, w1, w2 and w i Respectively represent the word weights corresponding to the 1st, 2nd and i-th keywords in the text.

[0080] It should be noted that the process of determining the word weight corresponding to each keyword is the same. The following will take the process of determining the word weight corresponding to any keyword as an example. Figure 4 The process of determining the word weight corresponding to any keyword is described in detail.

[0081] S303: Determine a word vector matrix of the first text according to the multiple keywords and the multiple word weights.

[0082] Optionally, for any one of the multiple keywords, a weighted word vector corresponding to the keyword can be determined based on the keyword and the word weight corresponding to the keyword, and the weighted word vector includes M-bit elements; multiple weighted word vectors corresponding to the multiple keywords are merged to obtain a weighted vector matrix of the first text; in the weighted vector matrix, the values ​​of the M-bit elements of the multiple weighted word vectors are added bit by bit to obtain a word vector matrix; wherein M is a positive integer.

[0083] It is understandable that when the amount of data of two texts to be compared for text similarity increases, the complexity of text calculation will increase, thereby reducing the efficiency of text similarity calculation. In view of this, the method can reduce the amount of data in the word vector matrix by merging and element-adding multiple weighted word vectors corresponding to multiple keywords, thereby helping to reduce the amount of data in subsequent text similarity comparison calculations and improve the efficiency of text similarity calculations.

[0084] In some embodiments, the word vector corresponding to the keyword can be obtained, and based on the word vector and word weight corresponding to the keyword, the weighted word vector corresponding to the keyword is determined. Assume that the word vector corresponding to the keyword is [f1…f1], where f1 and f1 represent the values ​​of the first and Mth elements of the word vector of the keyword respectively. The word weight corresponding to the keyword is w i , the weighted word vector corresponding to the keyword can be expressed as [f1×w i …f M × i ], where f1×w i and f M × i Represents the values ​​of the 1st and Mth elements of the weighted word vector.

[0085] Assuming there are n keywords, the weighted word vectors corresponding to these n keywords can be merged row by row. The weighted vector matrix after merging can be expressed as:

[0086]

[0087] Among them, w1, w2 and w n Respectively represent the word weights corresponding to the 1st, 2nd and nth keywords; f 11 、f 12 and f 1n Represents the value of the first element of the word vector of the first, second and nth keywords respectively; f M1 、f M2 and f Mn Represents the value of the word vector of the first, second and nth keywords at the Mth element; f 11 ×w1、f 12 ×w2 and f 1n × n Represent the values ​​of the 1st and Mth elements in the weighted word vectors of the 1st, 2nd, and nth keywords respectively.

[0088] In the weighted vector matrix, the values ​​of multiple elements with the same number of elements in multiple weighted word vectors can be summed to obtain the word vector matrix. For example, the values ​​of the first elements of multiple weighted word vectors can be summed to obtain the value of the first element of the word vector matrix.

[0089] In one example, the word embedding matrix can be expressed as:

[0090] [f 11 ×w1+f 12 ×w2+…+f 1n × n …f M1 ×w1+f M2 ×w2+…+f Mn × n ]

[0091] Among them, f 11 ×w1+f 12 ×w2+…+f 1n × n and f M1 ×w1+f M2 ×w2+…+f Mn × n They can represent the values ​​of the 1st and Mth elements of the word vector matrix respectively.

[0092] It should be noted that the process of determining the weighted word vector corresponding to each keyword is the same. The process of determining the weighted word vector corresponding to any keyword will be described as an example. Figure 5 The details are described in the examples.

[0093] S304: Normalize each element in the word vector matrix to obtain a first fingerprint.

[0094] Optionally, for any element in the word vector matrix, the element can be normalized in the following manner: if the value of the element is greater than 0, the value of the element is updated to the first value; if the value of the element is less than or equal to 0, the value of the element is updated to the second value.

[0095] Exemplarily, the first value may be 1, and the second value may be 0.

[0096] This method uses different normalization processing methods for each element in the word vector matrix based on the value of each element, which can avoid long similarity calculation time or calculation errors caused by large differences in the median values ​​of each element in the word vector matrix, and is conducive to improving the accuracy and efficiency of subsequent similarity calculations.

[0097] The text similarity determination method provided in the embodiment of the present application can determine the word weight corresponding to each keyword by combining the sentence component identifier and the part-of-speech identifier of each keyword, so as to further consider the influence of the word weight of each keyword, so as to improve the accuracy of determining the word weight of each keyword and the word vector matrix obtained based on the word weight calculation, which is beneficial to improving the accuracy of text similarity calculation; the method can also avoid long similarity calculation time or calculation errors due to excessive differences in the median values ​​of each element in the word vector matrix by normalizing the word vector matrix, which is beneficial to improving the accuracy and efficiency of subsequent similarity calculations.

[0098] On the basis of the above-mentioned embodiment, the text similarity determination method provided in the embodiment of the present application also involves a process of determining a plurality of word weights corresponding to a plurality of keywords. Figure 4 , taking the process of determining the word weight corresponding to any keyword as an example, the process of determining multiple word weights corresponding to multiple keywords is described in detail.

[0099] Figure 4 The third flow chart of the method for determining text similarity provided in the embodiment of the present application is as follows: Figure 4 As shown, the embodiment of the present application is Figure 2 Based on the embodiment, a method for determining text similarity is described in detail, and the method includes:

[0100] S401: Determine a first weight of the keyword according to the sentence component identifier and the part-of-speech identifier of the keyword.

[0101] Optionally, based on historical data of text similarity calculations of multiple groups of texts, sentence component weights corresponding to different sentence component identifiers and part-of-speech weights corresponding to different part-of-speech identifiers can be determined. The first weight is determined based on the sentence component weight corresponding to each sentence component identifier, the number of occurrences of each sentence component identifier, the part-of-speech weight corresponding to each part-of-speech identifier, and the number of occurrences of each part-of-speech identifier.

[0102] In some embodiments, the value range of the first weight can be set based on user business needs. After the first weight is calculated, if the first weight exceeds the value range, the calculated first weight can be normalized based on the value range. For example, the value range of the first weight f1 can be: 1<f1<2.

[0103] In one example, for sentence component identifiers with higher influence in text similarity calculation, the sentence component weights corresponding to these sentence component identifiers can be set larger; for sentence component identifiers with lower influence in text similarity calculation, the sentence component weights corresponding to these sentence component identifiers can be set smaller.

[0104] It can be understood that, assuming there are P sentence component identifiers, the sum of the P sentence component weights corresponding to the P sentence component identifiers may be a preset value, for example, the preset value may be 1 or 100%.

[0105] Similarly, for part-of-speech tags with higher influence in text similarity calculation, the part-of-speech weights corresponding to these part-of-speech tags can be set larger; for part-of-speech tags with lower influence in text similarity calculation, the part-of-speech weights corresponding to these part-of-speech tags can be set smaller.

[0106] It can be understood that, assuming there are Q part-of-speech identifiers, the sum of the Q part-of-speech weights corresponding to the Q part-of-speech identifiers may be a preset value, for example, the preset value may be 1 or 100%.

[0107] S402: Process the keywords based on multiple weight algorithms to obtain a second weight corresponding to the keywords under each weight algorithm.

[0108] The various weight algorithms may include a term frequency-inverse document frequency (TF-IDF) algorithm and a text ranking (TextRank) algorithm.

[0109] For the TF-IDF algorithm, the following formula can be used to determine the second weight corresponding to the keyword:

[0110]

[0111] Among them, F represents the second weight; TF represents the term frequency; IDF represents the inverse document frequency.

[0112] In some examples, when calculating the IDF of each keyword in the first text, the library may be the library where the first text is located. Assuming that the library includes the first text and the remaining T texts, the total number of documents in the library is T+1. Among these T+1 texts, if S texts include the keyword, the number of documents containing the keyword is S.

[0113] For the TextRank algorithm, the following formula can be used to determine the second weight corresponding to the keyword:

[0114]

[0115] Among them, W(V i ) can represent node V i The weight value; d represents the learning rate, for example, d can be set to 0.85; V i and V j Represents node i and node j respectively; In(V i) represents the node set consisting of all nodes pointing to node i; W ji Represents the weight of the edge from node i to node j.

[0116] It should be noted that the embodiments of the present application use the TF-IDF algorithm and the TextRank algorithm as examples to illustrate the weight algorithm. In some other examples, the multiple weight algorithms may also include any algorithm in the related technology for determining the importance of a keyword. The embodiments of the present application do not limit the number and type of weight algorithms.

[0117] S403: Determine a word weight according to the first weight, the algorithm weight of each weight algorithm, and multiple second weights corresponding to the keywords under multiple weight algorithms.

[0118] In some embodiments, the third weight of the keyword can be determined based on the algorithm weight of each weight algorithm and multiple second weights corresponding to the keyword under multiple weight algorithms; the word weight can be determined based on the first weight and the third weight.

[0119] It can be understood that, assuming there are R weight algorithms, the sum of the R algorithm weights corresponding to the R weight algorithms may be a preset value, for example, the preset value may be 1 or 100%.

[0120] When the first weight is f1, assume that there are two weight algorithms, among which the algorithm weight corresponding to the first weight algorithm is f2, the second weight of the keyword calculated based on the first weight algorithm is F, the algorithm weight corresponding to the second weight algorithm is f3, the second weight of the keyword calculated based on the second weight algorithm is W, and the word weight w of the keyword can be: w=f1×(f2×F+f3×W).

[0121] The text similarity determination method provided in the embodiment of the present application can comprehensively determine the word weight corresponding to each keyword by combining the sentence component identifier and part-of-speech identifier of each keyword and a variety of weight algorithms, so as to further improve the accuracy of determining the word weight corresponding to each keyword and the word vector matrix corresponding to the first text determined based on the word weight, thereby facilitating improving the accuracy of subsequent text similarity calculations.

[0122] Figure 5 A fourth flowchart of a method for determining text similarity provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the embodiment of the present application is Figure 2 Based on the embodiment, a method for determining text similarity is described in detail, and the method includes:

[0123] S501, performing hash mapping processing on the keyword to obtain a digital string corresponding to the keyword.

[0124] The number of digits in the digital string may be M, where M is a positive integer.

[0125] In some embodiments, the keyword may be vectorized to obtain an initial word vector corresponding to the keyword; the initial word vector may be mapped using a hash function to obtain an M-bit digital string.

[0126] The data format of the digital string may be in binary form, for example, an M-bit binary digital string is 1001…0110.

[0127] S502: Perform vector mapping processing on the digital string to obtain the word vector corresponding to the keyword.

[0128] A word vector may include M-bit elements, where M is a positive integer.

[0129] In some embodiments, the M-bit binary digital string can be mapped bit by bit to obtain the word vector corresponding to the keyword. For example, assuming that the M-bit binary digital string is 1001...0110, the space vector obtained by bit mapping is: [1 0 0 1...0 1 1 0].

[0130] Optionally, the word vector corresponding to the keyword can be a space vector with 1 row and M columns, and the space vector can be expressed as [f1…f M ], where f1 and f M can represent the values ​​of the 1st element and the Mth element of the space vector respectively.

[0131] In one example, for the i-th element f in the space vector i , f i It can be 1 or -1. i When it is 1, it means that the current binary position of the i-th element is 1; when f i When it is -1, it means that the current binary position of the i-th element is 0.

[0132] S503: According to the word weight corresponding to the keyword, weight the value of each element in the M-bit element of the word vector to obtain a weighted word vector corresponding to the keyword.

[0133] Assume that the word vector corresponding to the keyword is [f1…f1], where f1 and f1 represent the values ​​of the first and Mth elements of the word vector of the keyword respectively. The word weight corresponding to the keyword can be w i After weighting the value of each element of the word vector based on the word weight, the weighted word vector obtained can be expressed as [f1×w i …f M × i], where f1×w i and f M × i Represents the values ​​of the 1st and Mth elements of the weighted word vector.

[0134] The text similarity determination method provided in the embodiment of the present application can improve the accuracy of the keyword's word vector by converting each keyword into a word vector, and weighting the word vector in combination with the weight corresponding to each keyword to obtain the corresponding weighted word vector, thereby improving the accuracy of the subsequent calculation of text similarity based on the weighted word vector of the keyword.

[0135] Figure 6 The schematic diagram of the structure of the similarity determination device provided in this application is as follows: Figure 6 As shown, the similarity determination device 60 provided in this embodiment includes:

[0136] An extraction module 61 is used to perform summary extraction processing on the first text and the second text to obtain a first summary of the first text and a second summary of the second text;

[0137] The extraction module 61 is further used to perform text fingerprint extraction processing on the first text, the second text, the first summary and the second summary respectively to obtain a first fingerprint of the first text, a second fingerprint of the second text, a third fingerprint of the first summary and a fourth fingerprint of the second summary;

[0138] a determination module 62, configured to determine a first similarity based on the first fingerprint and the second fingerprint, and to determine a second similarity based on the third fingerprint and the fourth fingerprint;

[0139] The determination module 62 is further configured to determine the text similarity between the first text and the second text according to the first similarity and the second similarity.

[0140] In a possible implementation, the extraction module 61 may be specifically used for:

[0141] Preprocessing the first text to obtain a plurality of keywords of the first text, and a sentence component identifier and a part-of-speech identifier of each keyword;

[0142] Determining a plurality of word weights corresponding to the plurality of keywords according to the sentence component identifiers and part-of-speech identifiers of the plurality of keywords;

[0143] Determining a word vector matrix of the first text according to the multiple keywords and the multiple word weights;

[0144] Normalize each element in the word vector matrix to obtain the first fingerprint.

[0145] In a possible implementation, for any keyword, the extraction module 61 can also be specifically used to:

[0146] Determining a first weight of the keyword according to a sentence component identifier and a part-of-speech identifier of the keyword;

[0147] Processing the keyword based on multiple weight algorithms to obtain a second weight corresponding to the keyword under each weight algorithm;

[0148] The word weight is determined according to the first weight, the algorithm weight of each weight algorithm, and multiple second weights corresponding to the keywords under the multiple weight algorithms.

[0149] In a possible implementation, the extraction module 61 may also be specifically used for:

[0150] Performing word segmentation processing on the first text to obtain multiple candidate words;

[0151] Perform sentence component and part-of-speech tagging processing on each of the multiple candidate words to obtain a sentence component identifier and a part-of-speech identifier corresponding to each candidate word;

[0152] According to the sentence component identifier and the part-of-speech identifier corresponding to each candidate word, the multiple candidate words are filtered to obtain the multiple keywords and the sentence component identifier and the part-of-speech identifier of each keyword.

[0153] In a possible implementation, the extraction module 61 may also be specifically used for:

[0154] For any one of the multiple keywords, determine a weighted word vector corresponding to the keyword according to the keyword and a word weight corresponding to the keyword, wherein the weighted word vector includes M-bit elements;

[0155] Merging a plurality of weighted word vectors corresponding to the plurality of keywords to obtain a weighted vector matrix of the first text;

[0156] In the weighted vector matrix, performing element-by-element sum processing on the values ​​of the M-bit elements of the multiple weighted word vectors to obtain the word vector matrix;

[0157] Wherein, M is a positive integer.

[0158] In a possible implementation, the extraction module 61 may also be specifically used for:

[0159] Performing hash mapping processing on the keyword to obtain a digital string corresponding to the keyword, where the number of digits of the digital string is M;

[0160] Performing vector mapping processing on the digital string to obtain a word vector corresponding to the keyword, wherein the word vector includes M-bit elements;

[0161] According to the word weight corresponding to the keyword, the value of each element in the M-bit elements of the word vector is weighted to obtain a weighted word vector corresponding to the keyword.

[0162] In a possible implementation, for any element in the word vector matrix, the extraction module 61 can also be specifically used to:

[0163] If the value of the element is greater than 0, updating the value of the element to the first value;

[0164] If the value of the element is less than or equal to 0, the value of the element is updated to a second value.

[0165] In a possible implementation, the determination module 62 may be specifically configured to:

[0166] Determine text weight and summary weight;

[0167] Performing weighted processing on the first similarity according to the text weight to obtain a first weighted similarity;

[0168] performing weighted processing on the second similarity according to the summary weight to obtain a second weighted similarity;

[0169] The text similarity is obtained according to the first weighted similarity and the second weighted similarity.

[0170] The text similarity determination device provided in this embodiment can execute the method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0171] Figure 7 This is a schematic diagram of the structure of the text similarity determination device provided in this application. Figure 7 As shown, the text similarity determination device 70 provided in this embodiment includes: at least one processor 701 and a memory 702. Optionally, the text similarity determination device 70 also includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0172] In a specific implementation process, at least one processor 701 executes the computer-executable instructions stored in the memory 702, so that at least one processor 701 executes the above method.

[0173] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0174] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method involved in the above method embodiment.

[0175] An embodiment of the present application provides a computer program product, including a computer program, which implements the method involved in the above method embodiment when executed by a processor.

[0176] An embodiment of the present application provides a chip having a computer program stored thereon. When the computer program is executed by the chip, the method in the above method embodiment is implemented.

[0177] An embodiment of the present application provides a chip module, on which a computer program is stored. When the computer program is executed by the chip module, the method in the above method embodiment is implemented.

[0178] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0179] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0180] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0181] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0182] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not specifically stated, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not specifically stated, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0183] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0184] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0186] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for determining text similarity, characterized in that: The method comprises: Performing summary extraction processing on the first text and the second text to obtain a first summary of the first text and a second summary of the second text; Performing text fingerprint extraction processing on the first text, the second text, the first summary, and the second summary respectively to obtain a first fingerprint of the first text, a second fingerprint of the second text, a third fingerprint of the first summary, and a fourth fingerprint of the second summary; determining a first similarity based on the first fingerprint and the second fingerprint, and determining a second similarity based on the third fingerprint and the fourth fingerprint; The text similarity between the first text and the second text is determined according to the first similarity and the second similarity.

2. The method according to claim 1, characterized in that Performing text fingerprint extraction processing on the first text to obtain a first fingerprint of the first text includes: Preprocessing the first text to obtain a plurality of keywords of the first text, and a sentence component identifier and a part-of-speech identifier of each keyword; Determining a plurality of word weights corresponding to the plurality of keywords according to the sentence component identifiers and part-of-speech identifiers of the plurality of keywords; Determining a word vector matrix of the first text according to the multiple keywords and the multiple word weights; Normalize each element in the word vector matrix to obtain the first fingerprint.

3. The method according to claim 2, characterized in that For any keyword, determining the word weight corresponding to the keyword according to the sentence component identifier and part-of-speech identifier of the keyword includes: Determining a first weight of the keyword according to a sentence component identifier and a part-of-speech identifier of the keyword; Processing the keyword based on multiple weight algorithms to obtain a second weight corresponding to the keyword under each weight algorithm; The word weight is determined according to the first weight, the algorithm weight of each weight algorithm, and multiple second weights corresponding to the keywords under the multiple weight algorithms.

4. The method according to claim 2, characterized in that: Preprocessing the first text to obtain multiple keywords of the first text and a sentence component identifier and a part-of-speech identifier of each keyword includes: Performing word segmentation processing on the first text to obtain multiple candidate words; Perform sentence component and part-of-speech tagging processing on each of the multiple candidate words to obtain a sentence component identifier and a part-of-speech identifier corresponding to each candidate word; According to the sentence component identifier and the part-of-speech identifier corresponding to each candidate word, the multiple candidate words are filtered to obtain the multiple keywords and the sentence component identifier and the part-of-speech identifier of each keyword.

5. The method according to claim 2, characterized in that: Determining a word vector matrix of the first text according to the multiple keywords and the multiple word weights includes: For any one of the multiple keywords, determine a weighted word vector corresponding to the keyword according to the keyword and a word weight corresponding to the keyword, wherein the weighted word vector includes M-bit elements; Merging a plurality of weighted word vectors corresponding to the plurality of keywords to obtain a weighted vector matrix of the first text; In the weighted vector matrix, performing element-by-element sum processing on the values ​​of the M-bit elements of the multiple weighted word vectors to obtain the word vector matrix; Wherein, M is a positive integer.

6. The method according to claim 5, characterized in that Determining a weighted word vector corresponding to the keyword according to the keyword and the word weight corresponding to the keyword includes: Performing hash mapping processing on the keyword to obtain a digital string corresponding to the keyword, where the number of digits of the digital string is M; Performing vector mapping processing on the digital string to obtain a word vector corresponding to the keyword, wherein the word vector includes M-bit elements; According to the word weight corresponding to the keyword, the value of each element in the M-bit elements of the word vector is weighted to obtain a weighted word vector corresponding to the keyword.

7. The method according to claim 2, characterized in that For any element in the word vector matrix; Normalizing the elements in the word vector matrix includes: If the value of the element is greater than 0, updating the value of the element to the first value; If the value of the element is less than or equal to 0, the value of the element is updated to a second value.

8. The method according to any one of claims 1 to 7, characterized in that: Determining text similarity between the first text and the second text according to the first similarity and the second similarity includes: Determine text weight and summary weight; Performing weighted processing on the first similarity according to the text weight to obtain a first weighted similarity; performing weighted processing on the second similarity according to the summary weight to obtain a second weighted similarity; The text similarity is obtained according to the first weighted similarity and the second weighted similarity.

9. A device for determining text similarity, characterized in that: include: An extraction module, configured to perform summary extraction processing on the first text and the second text to obtain a first summary of the first text and a second summary of the second text; The extraction module is further configured to perform text fingerprint extraction processing on the first text, the second text, the first summary, and the second summary, respectively, to obtain a first fingerprint of the first text, a second fingerprint of the second text, a third fingerprint of the first summary, and a fourth fingerprint of the second summary; a determination module, configured to determine a first similarity according to the first fingerprint and the second fingerprint, and to determine a second similarity according to the third fingerprint and the fourth fingerprint; The determination module is further configured to determine the text similarity between the first text and the second text according to the first similarity and the second similarity.

10. A device for determining text similarity, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

12. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when being executed by a processor.

Images