A text matching method and system for long texts

Abstract

This invention discloses a text matching method and system for long texts. In this invention, the text pairs to be matched are input into a sentence-level filter to filter out noisy sentences and extract key sentences. Then, the key sentences are input into a word-level filter, and the deep interaction features between texts are mined using a BERT model incorporating the PageRank algorithm. Word-level noise filtering and fine-grained matching operations are performed on the key sentences. Finally, the relationship between text pairs is predicted by concatenating the vector representations of different positions in the BERT model. The positive effects of this invention are: (1) Compared with inputting all the contents of a long text into the model for training without deletion, deleting noisy sentences can effectively reduce the text length and remove useless information; (2) Deleting noisy words within BERT makes the model pay more attention to beneficial fine-grained matching signals, resulting in higher matching accuracy; (3) Combining the vector representations of different positions in the BERT output for the prediction task fully utilizes the semantic information encoded by the two texts, resulting in higher matching accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a text matching method and system for long texts. Background Technology

[0002] Text matching, a key task in natural language processing applications such as community question answering, information retrieval, and dialogue systems, aims to analyze and determine the semantic relationships between source and target texts. Long text matching is an important subfield of text matching, capable of quickly determining the relationship between two texts and identifying the similarity of their thematic expressions, thus possessing significant research and application value.

[0003] There are two main approaches to text matching models: traditional models and deep models. Traditional models involve manually defining and extracting features, and then using these extracted features to measure the degree of text matching. This method suffers from high costs and incomplete feature extraction. Furthermore, it is essentially a surface-level matching method and cannot achieve deeper semantic matching tasks.

[0004] The deep learning-based solution leverages the powerful language representation capabilities of deep neural networks to encode text, extract deep semantic information, and perform matching operations within the semantic space. This method eliminates the need for manually defined features and achieves high accuracy. Currently, most deep learning models are designed for short texts (i.e., short text deep matching models), while fine-grained matching signals between long texts are typically sparse. When using short text deep matching models to match long texts, it becomes difficult to identify matching signals from a large amount of noise, resulting in unsatisfactory matching results. Summary of the Invention

[0005] To overcome the aforementioned shortcomings of the prior art, the present invention provides a text matching method and system for long texts.

[0006] The technical solution adopted by this invention to solve its technical problem is: a text matching method for long texts, the text matching method comprising the following steps:

[0007] The two texts to be matched are input into a sentence-level filter, which extracts the corresponding key sentences for each text to be matched.

[0008] The key sentences of the text are input into the word-level filter, which performs word-level noise filtering and fine-grained matching operations, and outputs the filtered and matched text.

[0009] The BERT model outputs the position vectors corresponding to the two filtered and matched texts, and the position vectors are input into a 1D convolutional neural network to extract the semantic features of the text, thereby obtaining the feature representation vector of the comprehensive context of the two texts.

[0010] The feature representation vectors of the two texts and the feature representation vector of the CLS identifier in the BERT model are concatenated and input into a fully connected neural network to predict the similarity score. The similarity score is used as the basis for determining whether the texts match.

[0011] Furthermore, the step of extracting key sentences from the text using the sentence-level filter specifically includes:

[0012] Construct a graph model using the TextRank algorithm;

[0013] The graph model captures sentence-to-sentence similarity within a text and sentence-to-sentence similarity between texts;

[0014] Key sentences in the text are extracted based on the similarity between sentences within the text and the similarity between sentences between different texts.

[0015] Furthermore, the steps for constructing a graph model using the TextRank algorithm specifically include:

[0016] Source text and target text All sentences are input, where L1 and L2 represent the total number of sentences in the source and target texts, respectively. It refers to each sentence in the source text, the stated It refers to each sentence in the target text, where d s It is a set of source text sentences, the d t It is the set of target text sentences;

[0017] Combination d s and d t , obtain the set of all sentences

[0018] A graph model is constructed using sentences in S as vertices and the similarity between sentences as the weights of the edges.

[0019] Furthermore, the similarity between sentences is calculated as follows: the proportion of co-occurring words to the total number of words in the sentence is calculated, and sentence s... i and sentence s j Similarity between sim(s) i ,s j The calculation formula is shown in equation (1):

[0020]

[0021] The s i and s j Let i represent the i-th sentence and j-th sentence, respectively, and let sim(s) represent the i-th sentence and j-th sentence. i ,s j ) for si and s j Inter-similarity, the w k For in s i and s j Words that appear simultaneously in the text.

[0022] Furthermore, following the step of constructing the graph model using the TextRank algorithm, the following step is also included:

[0023] The TextRank algorithm is used to score sentences, and then key sentences are extracted based on the scores. i The rating value W(s) i The result is obtained by iterating through equation (2):

[0024]

[0025] The W(s) i ) is the sentence s i The weight value of W(s) j ) is the sentence s j The weight value, where d is the damping coefficient, representing the probability of pointing from a certain node in the graph to any other node, is typically set to 0.85. i s j and s k All are sentences from the sentence set, and the sim(s) i ,s j ) for s i and s j Inter-similarity, the sim(s) j ,s k ) for s j and s k Similarity between regions.

[0026] Furthermore, the word-level filter performs word-level noise filtering in the following steps:

[0027] The word-level filter is based on the BERT model and integrates the PageRank algorithm and Attention matrix to implement a word reduction strategy, filtering and deleting word-level noise information in the hidden layer.

[0028] Another object of the present invention is to provide a text matching system for long texts, the system comprising:

[0029] A sentence-level filter is used to receive two text inputs to be matched and extract the corresponding key sentences for each text to be matched.

[0030] A word-level filter is used to receive text key sentence input, perform word-level noise filtering and fine-grained matching operations on the text key sentence, and output the filtered and matched text.

[0031] The vector acquisition module is used to output the position vectors corresponding to the two filtered and matched texts using the BERT model, input the position vectors into a 1D convolutional neural network for text semantic feature extraction, and obtain the feature representation vector of the combined context of the two texts; and

[0032] The similarity analysis module is used to concatenate the feature representation vectors of two texts and the feature representation vector of the CLS identifier in the BERT model, and input them into a fully connected neural network to predict the similarity score. The similarity score is used as the basis for determining whether the texts match.

[0033] Furthermore, the sentence-level filter includes:

[0034] The graph model building module is used to build graph models using the TextRank algorithm.

[0035] A similarity capture module is used by the graph model to capture sentence-to-sentence similarity within a text and sentence-to-sentence similarity between texts; and

[0036] The key sentence extraction module is used to extract key sentences from the text based on the similarity between sentences within the text and the similarity between sentences between different texts.

[0037] Furthermore, the graph model construction module specifically includes:

[0038] The sentence input module is used to input source text. and target text All sentences are input; L1 and L2 represent the total number of sentences in the source text and target text, respectively. It refers to each sentence in the source text, the stated It refers to each sentence in the target text, where d s It is a set of source text sentences, the d t It is the set of target text sentences;

[0039] The sentence combination module is used to combine d s and d t , obtain the set of all sentences as well as

[0040] The graph model generation module is used to construct a graph model with sentences in S as vertices and the similarity between sentences as the weights of the edges.

[0041] In this invention, the text pairs to be matched are input into a sentence-level filter to filter out noisy sentences and extract key sentences. Then, the key sentences are input into a word-level filter, and the deep interaction features between texts are mined using a BERT model incorporating the PageRank algorithm. Word-level noise filtering and fine-grained matching operations are performed on the key sentences. Finally, the relationship between text pairs is predicted by concatenating the vector representations of different positions in the BERT model. The text matching method provided by this invention reduces noisy sentences and words in long texts and uses the simplified information for matching. Compared with the prior art, the positive effects of this invention are: (1) Compared with inputting all the contents of a long text into the model for training without deletion, reducing noisy sentences can effectively reduce the text length and remove useless information; (2) Reducing noisy words within BERT makes the model pay more attention to beneficial fine-grained matching signals, resulting in higher matching accuracy; (3) Combining the vector representations of different positions in the BERT output for prediction tasks fully utilizes the semantic information encoded by the two texts, resulting in higher matching accuracy. Attached Figure Description

[0042] The present invention will be described by way of example and with reference to the accompanying drawings, wherein:

[0043] Figure 1 This is a flowchart of a text matching method for long texts provided in an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram illustrating the principle of the text matching method provided in this embodiment of the invention;

[0045] Figure 3 This is a structural diagram of a text matching system for long texts provided in an embodiment of the present invention. Detailed Implementation

[0046] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings.

[0047] See Figure 1 and Figure 2 , Figure 1 The flowchart of the text matching method for long texts provided in the embodiments of the present invention is shown below:

[0048] In step S101, the two texts to be matched are input into a sentence-level filter, which extracts the corresponding key sentences for each text to be matched.

[0049] As an embodiment of the present invention, the step of extracting key sentences from text using the sentence-level filter specifically includes:

[0050] Construct a graph model using the TextRank algorithm;

[0051] The graph model captures sentence-to-sentence similarity within a text and sentence-to-sentence similarity between texts;

[0052] Key sentences in the text are extracted based on the similarity between sentences within the text and the similarity between sentences between different texts.

[0053] As an embodiment of the present invention, the step of constructing a graph model using the TextRank algorithm specifically includes:

[0054] Two texts and All sentences are input; L1 and L2 represent the total number of sentences in the source text and target text, respectively. It refers to each sentence in the source text, the stated It refers to each sentence in the target text, where d s It is a set of source text sentences, the d t It is the set of target text sentences;

[0055] Combination d s and d t , obtain the set of all sentences

[0056] A graph model is constructed using sentences in S as vertices and the similarity between sentences as the weights of the edges.

[0057] The calculation method for sentence similarity (including sentence similarity within a text and sentence similarity between texts) is as follows: calculate the proportion of co-occurring words to the total number of words in the sentence, and sentence s i and s j Similarity between sim(s) i ,s j The calculation formula is shown in equation (1):

[0058]

[0059] The s i and s j Let i represent the i-th sentence and j-th sentence, respectively, and let sim(s) represent the i-th sentence and j-th sentence. i ,s j ) for s i and s j Inter-similarity, the w k For in s i and s j Words that appear simultaneously in the text.

[0060] The step following the step of constructing the graph model using the TextRank algorithm also includes the following step:

[0061] The TextRank algorithm is used to score sentences, and then key sentences are extracted based on the scores. i The rating value W(s) i The result is obtained by iterating through equation (2):

[0062]

[0063] The W(s) i ) is the sentence s i The weight value of W(s) j ) is the sentence s j The weight value, where d is the damping coefficient, representing the probability of pointing from a certain node in the graph to any other node, is typically set to 0.85. i s j and s k All are sentences from the sentence set, and the sim(s) i ,s j ) for s i and s j Inter-similarity, the sim(s) j ,s k ) for s j and s k Similarity between regions.

[0064] In step S102, the key sentences of the text are input into the word-level filter, which performs word-level noise filtering and fine-grained matching operations, and outputs the filtered and matched text.

[0065] The specific steps for word-level noise filtering by the word-level filter include:

[0066] The word-level filter is based on the BERT model, and integrates the PageRank algorithm and the Attention matrix to implement a word reduction strategy, filtering and deleting word-level noise information in the hidden layer, thereby obtaining the filtered and matched text.

[0067] As an embodiment of the present invention, a word-level filter is used: key sentences of the text are input into the BERT model to mine fine-grained interactive semantic information between the two, and the attention matrix A and the PageRank algorithm are used to score the word nodes in the hidden layer of BERT. The scoring values ​​are divided into two parts. The first part first constructs a graph model on the hidden layer nodes of BERT, regards the attention matrix A as the adjacency matrix in PageRank, and then uses the PageRank algorithm to iterate. After convergence, the node importance value u is obtained, where the formula for the t-th iteration is shown in (3):

[0068] u t+1 =d(A l ) Tu t +(1-d) / N·Ι (3)

[0069] The u t Let A be the node weight after the t-th iteration. l Let be the attention matrix of the i-th layer, where d is the damping coefficient, N is the number of nodes in the graph, and I is the identity matrix.

[0070] Next, the attention matrix is ​​used as the weight matrix A, and multiplied with u to obtain the score value R = Au for the first part; for the second part, the matrix P obtained by summing the columns of matrix A is regarded as the initial importance score of the word node, and the attention matrix is ​​used as the weight matrix, and the two are multiplied to obtain the score value R under the effect of the initial value. * =AP. The two score values ​​are linearly summed to obtain the final score value R. final =αR * +(1-α)R, and R final Hidden nodes with lower scores are deleted based on this.

[0071] In step S103, the BERT model is used to output the position vectors corresponding to the two filtered and matched texts respectively. The position vectors are then input into a 1D convolutional neural network to extract the semantic features of the text, and the feature representation vector of the comprehensive context of the two texts is obtained.

[0072] In the output layer of the BERT model, the position vector H of the two texts is obtained by using the position of the SEP identifier as a separator. s =[h1,h2...,h SEP-1 ], H t =[h SEP+1 ,h SEP+2 ...,h N Then, the two texts are input into a 1D convolutional neural network to model the text context, resulting in feature representation vectors H for the two texts. s =Conv(H s H t =Conv(H t ).

[0073] In step S104, the feature representation vectors of the two texts and the feature representation vector of the CLS identifier in the BERT model are concatenated and input into the fully connected neural network to predict the similarity score. The similarity score is used as the basis for determining whether the texts match.

[0074] The feature representation of the CLS identifier in the BERT model H CLS The feature representations of the two texts H s H tThe concatenation is then input into the predicted similarity score in the two fully connected neural network layers:

[0075] socre = sigmoid(FC2(FC1([H CLS H s H t ]))) (4)

[0076] FC1 and FC2 are two fully connected neural networks, and sigmoid is the activation function.

[0077] Using the above method and system, input the following source text and target text. After the text matching method is applied, the output similarity score is 1 (that is, the source text and target text are similar).

[0078] Source text:

[0079] On September 14th, as Typhoon Muifa moved northwest, its main body began to affect Shanghai. In response to Typhoon Muifa, Shanghai Metro entered a state of full preparedness, issuing early warnings and responding rapidly. Around noon, at the Shanghai Railway Station station on Lines 3 and 4, station staff split into several groups to inspect flood control boards, check flood and typhoon prevention supplies, and confirm any leaks. Furthermore, since the platforms at Shanghai Railway Station on Lines 3 and 4 are open-air, station staff also secured movable items on the platforms, such as trash cans. Shanghai Metro stated that if the wind force reaches level 9, it will implement a suspension plan for the elevated lines. Reporters have just learned from Shanghai Metro that, in response to Typhoon Muifa, starting at 9 PM tonight, Shanghai Metro's ground and elevated lines will implement measures such as shortening routes or suspending operations to ensure the travel of citizens. Passengers are advised to travel in advance. If the typhoon's path or wind force changes, elevated and ground lines may also implement speed limits, shorten routes, suspend operations, or terminate operations earlier.

[0080] Target text:

[0081] In response to Typhoon Muifa, Shanghai Metro has entered a state of full readiness, issuing early warnings and responding rapidly. In the event of any emergency, train services will be suspended and passengers evacuated promptly to minimize the impact and losses from secondary disasters. Based on the current typhoon trend, starting at 9 PM tonight, Shanghai Metro's surface and elevated lines will implement measures such as shortened routes or suspensions to ensure the travel of citizens. Passengers are advised to travel in advance. If the typhoon's path or wind force changes, elevated and surface lines may also implement speed limits, shortened routes, suspensions, or termination of operations earlier than planned. Passengers are reminded that Lines 3, 5, 16, 17, the Pujiang Line, and the Maglev Line will be completely suspended, while Lines 1, 2, 4, 6, 7, 8, 9, 10, and 11 will be partially suspended. Furthermore, Shanghai Metro will adjust its operating schedule for tomorrow in real time based on the severity of the wind. The first train departure time for each line tomorrow morning may be delayed, and speed limits may be implemented during train operation, resulting in longer intervals between trains. For specific details, passengers are advised to follow the real-time information pushed by the official Weibo account of "Shanghai Metro" and the "Metro Metropolis" App, and adjust their travel routes accordingly.

[0082] Similarity (1 for similar / 0 for dissimilar): 1

[0083] See Figure 3 The diagram illustrates the structure of a text matching system for long texts provided in an embodiment of the present invention. The system includes: a sentence-level filter 31, a word-level filter 32, a vector acquisition module 33, and a similarity analysis module 34.

[0084] The sentence-level filter 31 receives two text inputs to be matched and extracts the corresponding key sentences for each text to be matched; the word-level filter 32 receives the key sentences and performs word-level noise filtering and fine-grained matching operations on the key sentences, outputting the filtered and matched text; the vector acquisition module 33 uses the BERT model to output the position vectors corresponding to the two filtered and matched texts, inputs the position vectors into a 1D convolutional neural network to extract text semantic features, and obtains the feature representation vector of the comprehensive context of the two texts; the similarity analysis module 34 concatenates the feature representation vectors of the two texts and the feature representation vector of the CLS identifier in the BERT model, inputs it into a fully connected neural network to predict the similarity score, and the similarity score serves as the basis for judging whether the texts match.

[0085] As an embodiment of the invention, the sentence-level filter 31 includes: a graph model construction module 311, a similarity capture module 312, and a key sentence extraction module 313.

[0086] The graph model construction module 311 constructs a graph model using the TextRank algorithm; the similarity capture module 312 captures the sentence similarity within the text and the sentence similarity between texts using the graph model; and the key sentence extraction module 313 extracts key sentences from the text based on the sentence similarity within the text and the sentence similarity between texts.

[0087] The graph model construction module 311 specifically includes: a sentence input module 3111, a sentence combination module 3112, and a graph model generation module 3113.

[0088] The sentence input module 3111 will input the source text. and target text All sentences are input; L1 and L2 represent the total number of sentences in the source text and target text, respectively. It refers to each sentence in the source text, the stated It refers to each sentence in the target text, where d s It is a set of source text sentences, the d t It is the set of target text sentences.

[0089] Sentence combination module 3112 combination d s and d t , obtain the set of all sentences The graph model generation module 3113 constructs a graph model using sentences in S as vertices and the similarity between sentences as the weights of the edges.

[0090] The word-level filter 32 performs word-level noise filtering as follows: Based on the BERT model, the word-level filter 32 integrates the PageRank algorithm and the Attention matrix to execute a word deletion strategy, and filters and deletes word-level noise information in the hidden layer.

[0091] In summary, this invention filters noisy sentences and extracts key sentences from the input sentence-level filter of the text pairs to be matched. Then, the key sentences are input into a word-level filter, and a BERT model incorporating the PageRank algorithm is used to mine deep interaction features between texts. Word-level noise filtering and fine-grained matching are then performed on the key sentences. Finally, the relationship between text pairs is predicted by concatenating vectors from different positions in the BERT model. The text matching method provided by this invention reduces noisy sentences and words in long texts and uses the simplified information for matching.

[0092] Compared with the prior art, the positive effects of the present invention are: (1) Compared with inputting all the contents of a long text into the model for training without deletion, deleting noisy sentences can effectively reduce the text length and remove useless information; (2) Deleting noisy words inside BERT makes the model pay more attention to beneficial fine-grained matching signals, resulting in higher matching accuracy; (3) Combining the vector representations at different positions in the BERT output for prediction tasks fully utilizes the semantic information after encoding the two texts, resulting in higher matching accuracy.

[0093] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A text matching method for long texts, characterized in that... The text matching method includes the following steps: The two texts to be matched are input into a sentence-level filter, which extracts the corresponding key sentences for each text to be matched. The key sentences of the text are input into the word-level filter, which performs word-level noise filtering and fine-grained matching operations, and outputs the filtered and matched text. The BERT model outputs the position vectors corresponding to the two filtered and matched texts, and the position vectors are input into a 1D convolutional neural network to extract the semantic features of the text, thereby obtaining the feature representation vector of the comprehensive context of the two texts. The feature representation vectors of the two texts and the feature representation vector of the CLS identifier in the BERT model are concatenated and input into a fully connected neural network to predict the similarity score. The similarity score is used as the basis for determining whether the texts match.

2. The text matching method for long texts according to claim 1, characterized in that, The specific steps for extracting key sentences from text using the sentence-level filter include: Construct a graph model using the TextRank algorithm; The graph model captures sentence-to-sentence similarity within a text and sentence-to-sentence similarity between texts; Key sentences in the text are extracted based on the similarity between sentences within the text and the similarity between sentences between different texts.

3. The text matching method for long texts according to claim 2, characterized in that, The steps for constructing a graph model using the TextRank algorithm specifically include: Source text and target text All sentences are input into a sentence-level filter, where L1 and L2 represent the total number of sentences in the source and target texts, respectively. It refers to each sentence in the source text, the stated It refers to each sentence in the target text, where d s It is a set of source text sentences, the d t It is the set of target text sentences; Combination d s and d t , obtain the set of all sentences A graph model is constructed using sentences in S as vertices and the similarity between sentences as the weights of the edges.

4. The text matching method for long texts according to claim 2, characterized in that, The similarity between sentences is calculated as follows: the proportion of co-occurring words to the total number of words in the sentence is calculated, and sentence S... i And sentence S j Similarity between sim(S) i ,S j The calculation formula is shown in equation (1): The S i and S j The sim(S) represents the i-th sentence and the j-th sentence, respectively. i ,S j ) is S i and S j Inter-similarity, the W k For in S i and S j Words that appear simultaneously in the text.

5. The text matching method for long texts according to claim 2, characterized in that, The step following the step of constructing the graph model using the TextRank algorithm also includes the following step: The TextRank algorithm is used to score sentences, and then key sentences are extracted based on the scores. The score W(si) of sentence si is obtained iteratively by equation (2): The W(S) i (S is a sentence) i The weight value of W(S) j (S is a sentence) j The weight value, where d is the damping coefficient, representing the probability of pointing from a certain node in the graph to any other node, is typically set to 0.

85. The S... i S j and S k All are sentences from the sentence set, and the sim(S) i ,S j ) is S i and S j The similarity between them, namely sim(S) j ,S k )) is S j and S k Similarity between regions.

6. The text matching method for long texts according to claim 1, characterized in that, The specific steps for word-level noise filtering by the word-level filter include: The word-level filter is based on the BERT model and integrates the PageRank algorithm and Attention matrix to implement a word reduction strategy, filtering and deleting word-level noise information in the hidden layer.

7. A text matching system for long texts, characterized in that... The system includes: A sentence-level filter is used to receive two text inputs to be matched and extract the corresponding key sentences for each text to be matched. A word-level filter is used to receive text key sentence input, perform word-level noise filtering and fine-grained matching operations on the text key sentence, and output the filtered and matched text. The vector acquisition module is used to output the position vectors corresponding to the two filtered and matched texts using the BERT model, input the position vectors into a 1D convolutional neural network for text semantic feature extraction, and obtain the feature representation vector of the combined context of the two texts; and The similarity analysis module is used to concatenate the feature representation vectors of two texts and the feature representation vector of the CLS identifier in the BERT model, and input them into a fully connected neural network to predict the similarity score. The similarity score is used as the basis for determining whether the texts match.

8. The text matching system for long texts according to claim 7, characterized in that, The sentence-level filter includes: The graph model building module is used to build graph models using the TextRank algorithm. A similarity capture module is used by the graph model to capture sentence-to-sentence similarity within a text and sentence-to-sentence similarity between texts; and The key sentence extraction module is used to extract key sentences from the text based on the similarity between sentences within the text and the similarity between sentences between different texts.

9. The text matching system for long texts according to claim 8, characterized in that, The graph model construction module specifically includes: The sentence input module is used to input source text. and target text All sentences are input into a sentence-level filter, where L1 and L2 represent the total number of sentences in the source and target texts, respectively. It refers to each sentence in the source text, the stated It refers to each sentence in the target text, where d s It is a set of source text sentences, the d t It is the set of target text sentences; Combination d s and d t , obtain the set of all sentences as well as The graph model generation module is used to construct a graph model with sentences in S as vertices and the similarity between sentences as the weights of the edges.

Images