A method and apparatus for calculating the similarity of long texts based on semantic progressive fusion

Abstract

This invention discloses a method and apparatus for calculating the similarity of long texts based on semantic progressive fusion, comprising: segmenting two long texts into several sentences to obtain sentence-level sequences and storing them in corresponding lists; repeatedly calling a Siamese BERT model to train several sentences in the lists to obtain the semantic representation of each sentence in the lists; training the sentence sequences of the long texts based on a Transformer model to obtain the semantic representations of the two long texts; and calculating the similarity between the two long texts based on the cosine formula of the angle between the vector spaces and the semantic representations of the two long texts. By constructing the semantic representation of long texts from a three-level structure of word-sentence-long text using the Siamese BERT model, the method more accurately captures the semantic information of long texts, improves the representation capability of semantic features, and solves the problems of input character limitations and missing sentence-level structural information in the native BERT model, significantly improving the accuracy of similarity calculation.

Description

Technical Field

[0001] This invention relates to the field of Chinese information processing technology, and in particular to a method and apparatus for calculating the similarity of long texts based on semantic progressive fusion. Background Technology

[0002] Text similarity calculation is a fundamental and hot research area in natural language processing, with wide applications in intelligent search, text mining, automatic question answering, and recommender systems. It possesses significant research value and enormous application prospects. Early techniques were based on statistical information about the text; these methods were simple in principle and easy to implement. However, early methods could not accurately understand the semantic information of the text.

[0003] With the maturity of deep learning technology, numerous neural network models have been used to calculate text similarity, achieving good results in sentence or short text similarity calculation tasks. In particular, the emergence of the pre-trained language model BERT in 2018 significantly improved the accuracy of NLP tasks through model pre-training and fine-tuning. However, these methods are not as effective with long texts. The main reason is that long texts are structurally more complex than sentences or short texts, making it difficult for existing methods to extract the richer semantic information hidden within the long text structure, resulting in unsatisfactory model performance. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for calculating the similarity of long texts based on semantic progressive fusion. It constructs the semantic representation of long texts from a three-level structure of words, sentences, and long texts using a Siamese BERT model. This model algorithm is suitable for calculating the similarity of long texts, enabling it to capture semantic information more accurately, improve the representation capability of semantic features, and provide more accurate and comprehensive semantic representation and inference for downstream tasks. It overcomes the problem of input character limitations in the native BERT model, effectively solving the problems of input character limitations and missing sentence-level structural information in the native BERT model, and significantly improving the accuracy of similarity calculation.

[0005] To address the aforementioned technical problems, a first aspect of this invention provides a method for calculating the similarity of long texts based on semantic progressive fusion, comprising the following steps:

[0006] The two long texts are each divided into several sentences, resulting in sentence-level sequences, which are then stored in the corresponding lists.

[0007] The Siamese BERT model is called repeatedly to train several statements in the list, thereby obtaining the semantic representation of each statement in the list.

[0008] The sentence sequence of the long text is trained based on the Transformer model to obtain two semantic representations of the long text.

[0009] The similarity between the two long texts is calculated based on the cosine formula of the angle between the two long texts and the semantic representation of the two long texts.

[0010] Furthermore, the iterative invocation of the Siamese BERT model to train several statements in the list, respectively, to obtain the semantic representation of each statement in the list, includes:

[0011] An improved Siamese BERT structure is used to capture the semantic dependencies between the statements, thereby obtaining semantic information of the statements containing more features. At the same time, based on a multi-head attention mechanism, multiple mappers are used to learn the representation features of different dimensions and subspaces in the statements, perform multi-feature fusion, and extract implicit information representations, ultimately obtaining the semantic representation of the statements with multi-feature fusion.

[0012] Furthermore, the iterative invocation of the Siamese BERT model to train several statements in the list, respectively, to obtain the semantic representation of each statement in the list, includes:

[0013] Based on the Bert model, dynamic word vectors that integrate contextual semantic information are obtained by training sentences at the word granularity using a whole-word mask.

[0014] Furthermore, the step of training the sentence sequence of the long text based on the Transformer model to obtain two semantic representations of the long text includes:

[0015] By analogy with the structural relationship between long texts and sentences, each of the statements in the list is used as a word-level token, and the long text is converted into a sentence level. The Transformer model is then used to train the sentence sequence of the long text to finally obtain a semantically progressive and fused text representation.

[0016] Accordingly, a second aspect of the present invention provides a long text similarity calculation apparatus based on semantic progressive fusion, comprising:

[0017] The text segmentation module is used to segment two long texts into several sentences, obtain sentence-level sequences, and store them into corresponding lists.

[0018] The statement training module is used to repeatedly call the Siamese BERT model to train several statements in the list and obtain the semantic representation of each statement in the list.

[0019] The long text training module is used to train the sentence sequence of the long text based on the Transformer model to obtain the semantic representations of the two long texts.

[0020] The similarity calculation module is used to calculate the similarity between the two long texts based on the cosine formula of the angle between the vector spaces and the semantic representation of the two long texts.

[0021] Furthermore, the statement training module uses an improved Siamese BERT structure to capture the semantic dependencies between statements, thereby obtaining semantic information of the statements containing more features. At the same time, based on a multi-head attention mechanism, it learns the representation features of different dimensions and subspaces in the statements through multiple mappers, performs multi-feature fusion, and extracts implicit information representations, ultimately obtaining the semantic representation of the statements with multi-feature fusion.

[0022] Furthermore, the sentence training module, based on the twin BERT model, trains sentences at the word granularity using a full-word mask to obtain dynamic word vectors that integrate contextual semantic information.

[0023] Furthermore, the long text training module, based on the structural relationship between long text and sentences, uses each statement in the list as a word-level token, converts the long text into sentence level, and uses the Transformer model to train the sentence sequence of the long text, ultimately obtaining a semantically progressive fusion text representation.

[0024] Accordingly, a third aspect of the present invention provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to cause the at least one processor to perform the above-described long text similarity calculation method based on semantic progressive fusion.

[0025] Accordingly, a fourth aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described long text similarity calculation method based on semantic progressive fusion.

[0026] The above-described technical solutions of the embodiments of the present invention have the following beneficial technical effects:

[0027] Based on the Siamese BERT model, this algorithm progressively constructs semantic representations of long texts from a three-level structure: word-sentence-long text. It is suitable for long text similarity calculation and can more accurately capture semantic information of long texts, improve the representation ability of semantic features, and provide more accurate and comprehensive semantic representations and inferences for downstream tasks. It overcomes the problem of input character limitations of the native BERT model and effectively solves the problems of input character limitations and missing sentence-level structural information in the native BERT model, and has a significant improvement in the accuracy of similarity calculation. Attached Figure Description

[0028] Figure 1 This is a flowchart of a long text similarity calculation method based on semantic progressive fusion provided in an embodiment of the present invention;

[0029] Figure 2 This is a flowchart of the long text similarity calculation model algorithm provided in the embodiments of the present invention;

[0030] Figure 3 This is an overall architecture diagram of the long text similarity calculation model provided in this embodiment of the invention;

[0031] Figure 4 This is a schematic diagram of the structure of the input layer of the Bert model provided in an embodiment of the present invention;

[0032] Figure 5 This is a schematic diagram of the long text semantic generation process provided in an embodiment of the present invention;

[0033] Figure 6 This is a vector representation and similarity diagram of the fifth group of long texts in the experimental data provided in this embodiment of the invention;

[0034] Figure 7 This is a block diagram of a long text similarity calculation device based on semantic progressive fusion provided in an embodiment of the present invention.

[0035] Figure label:

[0036] 1. Text segmentation module; 2. Sentence training module; 3. Long text training module; 4. Similarity calculation module. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments and the accompanying drawings. It should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0038] Please refer to Figure 1The first aspect of this invention provides a method for calculating the similarity of long texts based on semantic progressive fusion, comprising the following steps:

[0039] Step S100: Divide the two long texts into several sentences to obtain sentence-level sequences and store them into the corresponding lists.

[0040] Step S200: The Siamese BERT model is called repeatedly to train several statements in the list and obtain the semantic representation of each statement in the list.

[0041] Step S300: Train the sentence sequence of the long text based on the Transformer model to obtain the semantic representation of the two long texts.

[0042] Step S400: Based on the cosine formula of the angle between the vector spaces, calculate the similarity between the two long texts according to their semantic representations.

[0043] The technical solution of this invention employs a three-level structure based on the Siamese BERT model to progressively construct the semantic representation of long texts, from words to sentences to long texts. This model algorithm is suitable for long text similarity calculation and can more accurately capture the semantic information of long texts. Furthermore, a multi-head attention mechanism is used to obtain the semantic representation of the entire sentence. Simultaneously, to better identify the importance of different components within the sentence and obtain a sentence representation that incorporates more feature information, a multi-head attention mechanism is added. This mechanism uses multiple mappers to learn representation features from different dimensions and subspaces. Finally, multiple features are fused to extract more important implicit information representations as the final sentence semantic vector.

[0044] Specifically, by transforming long texts into a sequence of sentence vectors with a sequential order, the long text becomes the role of sentences, and the sentences become the role of words. The sentence vector sequence is used to replace the word vector sequence in the original model for training, thereby obtaining a semantic representation of long texts that integrates contextual information and hidden internal text features.

[0045] The semantic representation of long texts is mainly based on the characteristics of the grammatical structure of long texts. It adopts a three-level structure from words to sentences to text to progressively construct the semantic representation of long texts. Information representations are generated at different levels and used as inputs for the next level. That is, sentence semantic representations are generated by obtaining word vectors, and then text semantic representations are generated by obtaining sentence semantic representations. Finally, a semantically progressive and fused text representation is obtained.

[0046] The three key process points in the above steps include: dynamic word vector generation, sentence vector generation containing word order information and enhanced semantic features, and long text semantic vector generation.

[0047] Specifically, step S200 involves repeatedly calling the Siamese BERT model to train several statements in the list and obtain the semantic representation of each statement in the list. This further includes the following steps:

[0048] An improved Siamese BERT structure is used to capture semantic dependencies between statements, resulting in semantic information of statements containing more features. At the same time, based on a multi-head attention mechanism, multiple mappers are used to learn the representation features of different dimensions and subspaces in the statements, perform multi-feature fusion, and extract implicit information representations, ultimately obtaining the semantic representation of the statements with multi-feature fusion.

[0049] Furthermore, step S200, which involves repeatedly calling the Siamese BERT model to train several statements in the list and obtain the semantic representation of each statement in the list, further includes the following steps:

[0050] Based on the BERT model, dynamic word vectors that integrate contextual semantic information are obtained by training sentences at the word granularity using a whole-word mask.

[0051] Further, in step S300, the Transformer model is used to train the sentence sequence of the long text to obtain the semantic representations of the two long texts, including:

[0052] By analogy with the structural relationship between long texts and sentences, each statement in the list is used as a word-level token, and the long text is converted into a sentence level. The Transformer model is then used to train the sentence sequence of the long text, and finally a semantically progressive fusion text representation is obtained.

[0053] Please refer to Figure 2 As shown in the overall architecture of the long text similarity calculation model,

[0054] Considering the grammatical structure of long Chinese texts, a three-tiered semantic representation is generated progressively from words to sentences to text. Each level generates corresponding information representations as input for the next level. Specifically, first, sentence semantic representations are generated by obtaining word vectors, and then text semantic representations are generated from these sentence semantic representations. Finally, based on the spatial mapping of the long text semantic vectors, the similarity between two long texts is calculated using the cosine of the angle between them. The algorithm flow of the model architecture is as follows:

[0055] (1) Input two long texts and split each long text into individual sentences, storing them in two lists in the order of the sentences.

[0056] (2) Repeatedly call the Bert model to obtain the semantic representation of each statement in the list.

[0057] In this process, firstly, dynamic word vectors that can accurately integrate contextual semantic information are obtained through BERT feature extraction. At the same time, the previous approach of generating sentence representations by simple linear combination is abandoned. Based on the Siamese BERT model, the semantic representation of long text is constructed progressively from a three-level structure of word-sentence-long text. The model algorithm is suitable for long text similarity calculation and can capture the semantic information of long text more accurately.

[0058] (3) The long text is converted into a sentence-level sequence, and the semantic representation of the long text is obtained through training.

[0059] Analogous to the structural relationship between long text and sentences, long text is composed of sentences, and sentences are composed of words. If a model is trained on the word sequence of a sentence, then it can naturally be trained on the sentence sequence of a text. Therefore, if each sentence in the list is regarded as a word-level token, then a long text is transformed into sentences at another level. The Transformer model is then used again to train the sentence sequence of the long text, and finally a semantically progressive and fused text representation is obtained.

[0060] (4) After obtaining the semantic representations of the two long texts, the similarity value between them is calculated using the cosine formula of the angle between the vector spaces.

[0061] Please refer to Figure 3 The BERT model obtains the vector representation of a sentence through an input phase and a Transformer encoding phase. The input phase extracts three important vectors to represent the information of the input sentence: word vectors, segmentation vectors, and position vectors. Word vectors represent the vector corresponding to each token in the sentence. These vectors are pre-trained by the BERT model, containing semantic and contextual information for each token in the input sequence, and can adaptively adjust based on subsequent encoding and training. Segmentation vectors correspond to the NSP (Next Sentence Prediction) task in the BERT model, assigning a unique identifier to each sentence to distinguish between two different sentences in the input text. Position vectors supplement the lack of positional information in the Transformer model by artificially adding vectors representing positions, enabling the model to understand the positional relationships of each word in the text and thus identify the semantic connections between words. These three vectors are superimposed to form the initial input vector of the BERT model.

[0062] After the input phase, the BERT model uses a multi-layer Transformer encoder to process the input vector, thereby capturing various semantic relationships in the text and generating corresponding output vectors. Furthermore, considering that different words in a sentence have varying importance in conveying the overall information, this study adds a multi-head attention mechanism after generating the sentence vector to extract more important latent information representations as the final semantic representation of the sentence. Multi-head attention learns representational features from different dimensions and subspaces by using multiple mappers, and finally fuses these features, which further improves feature representation capabilities compared to a single attention mechanism.

[0063] Please refer to Figure 4 Considering the original BERT model's limitation on the length of word sequences within a sentence (maximum 512 characters, including [CLS] and [SEP]), this is clearly insufficient for long texts. Therefore, drawing an analogy to the structural relationship between long texts and sentences, we consider using sentence sequences instead of word sequences to retrain and obtain the semantic representation of the long text. Using the previously obtained sentence vectors as input vectors to the Transformer encoder, we simulate BERT's method of generating token-level vectors to train the sentence vectors, thereby acquiring complex information such as the structure between sentences in the long text. Assuming a long text is divided into N sentences according to sentence termination symbols, the N sentence vectors obtained after training with the BERT model are stored in a list. The Transformer model's positional encoder processes these vectors so that the model can combine the sentence-level structure of the long text to train the sentence vectors, broadly capturing the structural relationships and semantic information of each sentence in the entire sequence, thus improving the text vectors' ability to express the semantic information of the text. After model training, the long text is mapped to a vector in space, and this vector representation can effectively integrate text information and express text semantics.

[0064] For example, if we define the maximum length of words in a sentence as n, the shape of the result obtained from such a sentence would be (n, 768). Assuming a sample contains 80 sentences, if there are more than 80 sentences, the first 80 are taken; if there are fewer than 80 sentences, the 768-dimensional vector is filled with zeros. The resulting vector would then be (N, 80, 768), where N represents the number of samples, 80 represents the maximum sentence length, and 768 represents the vector dimension. Finally, the Transformer model is used to obtain a vector representation that incorporates the semantic information of the entire long text, including all sentences.

[0065] Table 1 shows a comparison of experimental data between the algorithm of this invention and the original BERT model in similarity calculation. In the table, Label represents the text similarity value determined by experts (1 indicates similarity, 0 indicates dissimilarity), Similarity represents the text similarity value calculated by the model of this invention, and Probability represents the text similarity value calculated by the original BERT model. The long text similarity calculation model proposed in this invention achieves better classification performance in long text dataset processing, more realistically representing the semantic information of long texts and effectively improving the accuracy of long text semantic similarity. Taking the second and third groups of data in the experiment as examples, the experimental results match the actual labels, and the similarity of the proposed model is more closely aligned with the text content and has greater discriminative power. The cosine similarity calculated by this model for the second group is 0.8095, consistent with the expert judgment result; when the text label is 0, the cosine similarity calculated by the model for the third group is 0.0347, and the model classifies these two long texts as dissimilar, with the experimental results matching the actual labels well. Furthermore, this model is better suited for semantic similarity calculation tasks involving long texts, as it can establish closer relationships between text fragments at the semantic level. Compared to the native BERT model, this patented model can more accurately obtain semantic similarity information for long texts. Taking the first and fifth sets of data in the experiment as examples, when two texts have some slight semantic relationships but are not enough to be classified as similar, the model in this paper can better describe the semantic relationships between the texts. For example, the similarity score calculated by the BERT model for the first set is 0.0014, while the similarity score calculated by the model in this paper is 0.6756; the similarity score calculated by the BERT model for the fifth set is 0.0017, while the similarity score calculated by the model in this paper is 0.5224. Both sets of data accurately indicate that the two long texts are similar in some aspects but have significant differences in others.

[0066] Table 1

[0067]

[0068]

[0069] Furthermore, in step S400, the similarity between two texts is determined based on cosine similarity. The cosine of the angle between two vectors in high-dimensional space is used as the similarity criterion, with a value ranging from -1 to 1. The value is independent of the lengths of the two vectors, primarily depending on their directions. When the two vectors point in the same direction, the cosine similarity is 1; otherwise, it is -1. Let the vector representations of the two texts be A and B, then the similarity between the two texts is calculated as follows:

[0070]

[0071] Specifically, as shown in Table 2, directly using the [CLS] output vector from BERT as the sentence representation does not yield very good results. Given BERT's advantage in obtaining dynamic, context-aware, token-level vector representations, this invention utilizes BERT modeling to obtain dynamic word vectors rich in contextual information. Then, the token-level word vector representations are processed through a more suitable network model to obtain sentence vectors containing richer information. Furthermore, when pre-training a language model, BERT requires masking the corpus, enabling the model to predict masked words without seeing the tokens. In the BERT model, Chinese is segmented at the character level, without considering the need for word segmentation. A Chinese BERT model applying full-word masking should perform better. The full-word masking method was applied to Chinese text, [MASKing] all Chinese characters that make up the same word. Compared to the original BERT Chinese model and the open-source Chinese pre-trained model ERNIE, this model achieved better performance in multiple Chinese tasks. Full-word masking requires first segmenting the Chinese corpus into words, and then masking each token belonging to the same word. In this way, the pre-trained model does not predict the individual masked tokens, but rather every masked token within the same word. Therefore, by using a whole-word masking approach, the given long text is first segmented into Chinese words, and then fed into the BERT model for training to obtain dynamic word vectors for each word.

[0072] Table 2

[0073]

[0074] Accordingly, please refer to Figure 7 A second aspect of the present invention provides a long text similarity calculation device based on semantic progressive fusion, comprising:

[0075] Text segmentation module 1 is used to segment two long texts into several sentences, obtain sentence-level sequences, and store them into corresponding lists;

[0076] Statement training module 2 is used to repeatedly call the Siamese BERT model to train several statements in the list and obtain the semantic representation of each statement in the list.

[0077] Long text training module 3 is used to train the sentence sequence of long text based on the Transformer model to obtain the semantic representation of two long texts;

[0078] Similarity calculation module 4 is used to calculate the similarity between two long texts based on the cosine formula of the angle between the vector spaces and the semantic representation of the two long texts.

[0079] Furthermore, the sentence training module 2 uses an improved Siamese BERT structure to capture the semantic dependencies between sentences, thereby obtaining semantic information of sentences containing more features. At the same time, based on the multi-head attention mechanism, it learns the representation features of different dimensions and different subspaces in the sentences through multiple mappers, performs multi-feature fusion, and extracts implicit information representations, ultimately obtaining the semantic representation of the sentences with multi-feature fusion.

[0080] Furthermore, the sentence training module 2 is based on the Siamese BERT model. It trains sentences at the word granularity using a whole-word mask to obtain dynamic word vectors that integrate contextual semantic information.

[0081] Furthermore, the long text training module 3, based on the structural relationship between long text and sentences, uses each statement in the list as a word-level token and converts the long text into a sentence level. It then uses the Transformer model to train the sentence sequence of the long text, ultimately obtaining a semantically progressive fusion text representation.

[0082] Accordingly, a third aspect of the present invention provides an electronic device, including: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions executable by a processor, the instructions being executed by the processor to cause the at least one processor to perform the above-described long text similarity calculation method based on semantic progressive fusion.

[0083] Accordingly, a fourth aspect of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described long text similarity calculation method based on semantic progressive fusion.

[0084] This invention aims to protect a method and apparatus for calculating the similarity of long texts based on semantic progressive fusion, comprising the following steps: dividing two long texts into several sentences to obtain sentence-level sequences and storing them in corresponding lists; repeatedly calling the Siamese BERT model to train several sentences in the lists to obtain the semantic representation of each sentence in the lists; training the sentence sequences of the long texts based on the Transformer model to obtain the semantic representations of the two long texts; and calculating the similarity between the two long texts based on the cosine formula of the angle between the vector spaces and the semantic representations of the two long texts. The above technical solution has the following effects:

[0085] Based on the Siamese BERT model, this algorithm progressively constructs semantic representations of long texts from a three-level structure: word-sentence-long text. It is suitable for long text similarity calculation and can more accurately capture semantic information of long texts, improve the representation ability of semantic features, and provide more accurate and comprehensive semantic representations and inferences for downstream tasks. It overcomes the problem of input character limitations of the native BERT model and effectively solves the problems of input character limitations and missing sentence-level structural information in the native BERT model, and has a significant improvement in the accuracy of similarity calculation.

[0086] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for calculating the similarity of long texts based on semantic progressive fusion, characterized in that, Includes the following steps: The two long texts are each divided into several sentences, resulting in sentence-level sequences, which are then stored in the corresponding lists. The Siamese BERT model is called repeatedly to train several statements in the list, thereby obtaining the semantic representation of each statement in the list. The sentence sequence of the long text is trained based on the Transformer model to obtain two semantic representations of the long text. Based on the cosine formula of the angle between the two long texts, the similarity between the two long texts is calculated according to the semantic representation of the two long texts; The loop calls the Siamese BERT model to train several statements in the list, obtaining the semantic representation of each statement in the list, including: An improved Siamese BERT structure is used to capture the semantic dependencies between the statements, thereby obtaining semantic information of the statements containing more features. At the same time, based on a multi-head attention mechanism, multiple mappers are used to learn the representation features of different dimensions and subspaces in the statements, perform multi-feature fusion, and extract implicit information representations, ultimately obtaining the semantic representation of the statements with multi-feature fusion.

2. The long text similarity calculation method based on semantic progressive fusion according to claim 1, characterized in that, The loop calls the Siamese BERT model to train several statements in the list, obtaining the semantic representation of each statement in the list, including: Based on the Bert model, dynamic word vectors that integrate contextual semantic information are obtained by training sentences at the word granularity using a whole-word mask.

3. The long text similarity calculation method based on semantic progressive fusion according to claim 1 or 2, characterized in that, The process of training the sentence sequence of the long text using the Transformer model yields two semantic representations of the long text, including: By analogy with the structural relationship between long texts and sentences, each of the statements in the list is used as a word-level token, and the long text is converted into a sentence level. The Transformer model is then used to train the sentence sequence of the long text to finally obtain a semantically progressive and fused text representation.

4. A long text similarity calculation device based on semantic progressive fusion, characterized in that, include: The text segmentation module is used to segment two long texts into several sentences, obtain sentence-level sequences, and store them into corresponding lists. The statement training module is used to repeatedly call the Siamese BERT model to train several statements in the list and obtain the semantic representation of each statement in the list. The long text training module is used to train the sentence sequence of the long text based on the Transformer model to obtain the semantic representations of the two long texts. The similarity calculation module is used to calculate the similarity between the two long texts based on the cosine formula of the angle between the vector spaces and the semantic representation of the two long texts. The statement training module uses the improved Siamese BERT structure to capture the semantic dependencies between statements, thereby obtaining semantic information of the statements containing more features. At the same time, based on the multi-head attention mechanism, it learns the representation features of different dimensions and different subspaces in the statements through multiple mappers, performs multi-feature fusion, and extracts implicit information representations, finally obtaining the semantic representation of the statements with multi-feature fusion.

5. The long text similarity calculation device based on semantic progressive fusion according to claim 4, characterized in that, The sentence training module, based on the twin BERT model, trains sentences at the word granularity using a full-word mask to obtain dynamic word vectors that integrate contextual semantic information.

6. The long text similarity calculation device based on semantic progressive fusion according to claim 4 or 5, characterized in that, The long text training module is based on the structural relationship between long text and sentences. It takes each statement in the list as a word-level token, converts the long text into a sentence level, and uses the Transformer model to train the sentence sequence of the long text, finally obtaining a semantically progressive fusion text representation.

7. An electronic device, characterized in that, include: At least one processor; And a memory connected to the at least one processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to cause the at least one processor to perform the long text similarity calculation method based on semantic progressive fusion as claimed in any of claims 1-3.

8. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement any of the long text similarity calculation methods based on semantic progressive fusion as described in claims 1-3.

Images