A long text compression method and device based on a language model

Abstract

The embodiment of the application discloses a long text compression method and device based on a language model, wherein the method comprises the following steps: performing sentence division on a first initial long text by a regular method to obtain a first sentence sequence, and performing sentence division on a second initial long text by the regular method to obtain a second sentence sequence; obtaining a third sentence sequence from the first sentence sequence according to a Top K algorithm, and obtaining a fourth sentence sequence from the second sentence sequence according to the Top K algorithm; determining a sixth sentence sequence with the maximum similarity to the third sentence sequence from a fifth sentence sequence, and determining an eighth sentence sequence with the maximum similarity to the fourth sentence sequence from a seventh sentence sequence; splicing the third sentence sequence and the eighth sentence sequence to obtain a ninth sentence sequence, and splicing the fourth sentence sequence and the sixth sentence sequence to obtain a tenth sentence sequence. The application can improve the accuracy of long text compression.

Description

Technical Field

[0001] This application relates to Internet technology and is applied in fields such as smart devices, artificial intelligence, and law, and in particular to a long text compression method and apparatus based on a language model. Background Technology

[0002] In the legal industry, legal judgments are often lengthy, sometimes reaching tens of thousands of words. Therefore, effective processing of long texts is crucial for case retrieval and matching tasks to improve the accuracy of long text retrieval and matching models. Existing long text processing methods include truncation and model-based approaches. Truncation methods include: ① Extracting the beginning and end of sentences: The beginning and end of sentences contain a significant amount of information and can be extracted and concatenated; ② Random truncation: Extracting the beginning and end of sentences may result in significant information loss, so sentences can be randomly extracted from the long text to expand the dataset. This method allows the model to learn more diverse sample types; ③ Extracting the beginning or end of sentences, and then extracting keywords from the remaining sentences and concatenating them to the extracted sentences; ④ Dividing the long text into multiple samples using random truncation or a fixed sliding window, and averaging the results during prediction as the final matching value. However, the random truncation and sliding window averaging methods in the truncation method fail to capture the important information of long texts. This method assumes that each sentence is equally important, which leads to insufficient generalization of the model. On the other hand, the method of truncating the beginning and end of sentences will lose too much information. Finally, truncating the beginning or end of sentences and extracting the remaining sentences through keyword extraction depends on the accuracy of the keyword extraction method.

[0003] In addition, model-based methods include: ① Attention mechanism: dividing long texts into several sentences and introducing an attention mechanism to learn the importance of each sentence, further learning the features of the long text; ② Text summarization: training a summarization generation model to generate summaries of long texts, which are then used as the training set. The drawbacks of model-based methods are: while attention-based methods can learn key sentences to some extent and have a certain degree of generalization, the overall model is relatively complex with too many parameters, which may not meet the response requirements of immediate applications. Furthermore, the method of generating text summaries is uncontrollable and highly dependent on the generalization ability of the generation model. Summary of the Invention

[0004] This application provides a language model-based method and apparatus for compressing long texts, which can improve the accuracy of long text compression results.

[0005] In a first aspect, embodiments of this application provide a long text compression method based on a language model, the method comprising:

[0006] The initial long text is segmented into sentences using regular expressions to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S nThe second initial long text is segmented into sentences using the regular expression method to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ], wherein there is a correlation between the first initial long text and the second initial long text;

[0007] According to the Top K algorithm, from the first sentence sequence [S1, S2, ..., S...] n-1 S n Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k ], and according to the Top K algorithm from the second sentence sequence [T1, T2, ..., T m-1 T m Extract the fourth sentence sequence [T1, T2, ..., T] from the data. k-1 T k ];

[0008] Determine the sequence from the fifth sentence sequence that corresponds to the third sentence sequence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ], wherein the fifth sentence sequence is the first sentence sequence [S1, S2, ..., S... n-1 S n The remaining sentence sequence after truncation in ] [S k+1 S k+2 S n-1 S n The seventh sentence sequence is the second sentence sequence [T1, T2, ..., T]. m-1 T m The remaining sentence sequence after truncation in [T] k+1 T k+2 ,…,T m-1 T m ];

[0009] The third sentence sequence [S1, S2, ..., S...] is used to... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D nBy concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n ].

[0010] In existing technologies, truncation methods can lead to insufficient model generalization or reliance on the accuracy of keyword extraction methods. Model-based methods, on the other hand, rely on complex models with too many parameters, potentially failing to meet the responsiveness requirements of real-time applications. Furthermore, the method of generating text summaries is uncontrollable and highly dependent on the generalization of the generated model. This application primarily uses a Jelinek-Mercer smoothing language model to compress long texts. Specifically, regularization methods and open-source tools are used to segment two initial long texts (comprising 200 sentences and 10,000 characters in total) into sentences and words, resulting in the first sentence sequence [S1, S2, ..., S...]. n-1 S n (Including 20 sentences totaling 800 characters) and the second sentence sequence [T1, T2, ..., T... m-1 T m [(Including 20 sentences with a total of 800 characters), then using the TopK algorithm to extract the first sentence sequence [S1, S2, ..., S...] n-1 S n The second sentence sequence [T1, T2, ..., T] m-1 T m Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k [(Including 5 sentences totaling 200 characters) and the fourth sentence sequence [T1, T2, ..., T] k-1 T k (Including 5 sentences totaling 200 characters), then based on the fifth sentence sequence remaining after extracting from the first sentence sequence [S] k+1 S k+2 S n-1 S n (Including 15 sentences totaling 600 characters) Determine the sequence [S1, S2, ..., S] of the third sentence. k-1 S kThe sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and according to the sequence from the second sentence [T1, T2, ..., T m-1 T m The remaining seventh sentence sequence after truncation [T] k+1 T k+2 ,…,T m-1 T m Determine the sequence of the fourth sentence [T1, T2, ..., T] k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n Finally, the third sentence sequence [S1, S2, ..., S...] is... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n The sequence [T1, T2, ..., T] is concatenated, and the fourth sentence sequence [T1, T2, ..., T] is combined. k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n The compressed text is obtained by concatenating the words. This application is based on a Jelinek-Mercer smoothing optimized language model, which makes the probability distribution of words in the language model closer to reality, thereby improving the accuracy of long text compression results.

[0011] In one possible implementation, the first initial long text is segmented into sentences using a regular expression method to obtain a first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text is segmented into sentences using the regular expression method to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ],include:

[0012] The first initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the first sentence sequence [S1, S2, ..., S]. n-1 S n The second initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ].

[0013] In the above method, the main steps of data cleaning using regular expressions and the open-source tool Jieba are: word segmentation, stop word removal preprocessing, and punctuation removal preprocessing for stop words and punctuation marks in the first and second initial long texts. Specifically, for example, if the sentence endings in the first initial long text contain three types of punctuation marks: ":", ",", and ".", the server can segment the first initial long text according to these three punctuation mark types to obtain the first sentence sequence. If the first initial long text is "She said: I will handle this matter, don't worry.", then the server can segment the first initial long text into "She said", "I will handle this matter", and "Don't worry". This application utilizes regular expressions and the open-source tool Jieba to process long text strings, enabling faster and more concise matching of target string content and string processing operations such as retrieval and replacement of target objects.

[0014] In another possible implementation, the first initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ],include:

[0015] The first initial long text is segmented into characters to obtain multiple characters corresponding to the first initial long text, and the second initial long text is segmented into characters to obtain multiple characters corresponding to the second initial long text;

[0016] Encode each character among the multiple characters corresponding to the first initial long text to obtain a first feature vector of each character among the multiple characters corresponding to the first initial long text, and encode each character among the multiple characters corresponding to the second initial long text to obtain a second feature vector of each character among the multiple characters corresponding to the second initial long text, wherein the first feature vector and the second feature vector are used to represent the context information of each character among the multiple characters;

[0017] Based on the first feature vector and the language type of the first initial long text, determine the first sentence sequence [S1, S2, ..., S...]. n-1 S n ], and based on the second feature vector and the language type of the second initial long text, determine the second sentence sequence [T1, T2, ..., T] m-1 T m ].

[0018] In the above method, the specific steps of the server performing word segmentation, stop word removal, and punctuation removal preprocessing on the first and second initial long texts can be as follows: First, the two initial long texts are segmented into characters to obtain multiple characters corresponding to the initial long texts. Then, each character in the multiple characters corresponding to the initial long texts is encoded to obtain a feature vector. Finally, the first sentence sequence [S1, S2, ..., S...] is determined based on the feature vectors of the two initial long texts and the corresponding language types. n-1 S n The second sentence sequence [T1, T2, ..., T] m-1 T m (For example, the above steps are the specific process of performing regular expression processing on a first initial long text containing 200 sentences and 10,000 characters to obtain a first sentence sequence containing 20 sentences and 800 characters, and performing regular expression processing on a second initial long text containing 200 sentences and 10,000 characters to obtain a second sentence sequence containing 20 sentences and 800 characters.) This application uses regular expression methods and the open-source tool Jieba to process long text strings, which can effectively improve the efficiency of long text compression.

[0019] In yet another possible implementation, the step of determining from the fifth sentence sequence the sequence of sentences related to the third sentence sequence [S1, S2, ..., S...] k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ],include:

[0020] Based on the remaining fifth sentence sequence after truncation [S] k+1 S k+2 S n-1 S n Construct a unigram language model M1, and based on the remaining seventh sentence sequence [T] after truncation. k+1 T k+2 ,…,T m-1 T m Construct a unigram language model M2;

[0021] According to the Top N algorithm, the third sentence sequence [S1, S2, ..., S...] is used to... k-1 S kThe input is fed into the unigram language model M2 to obtain the sequence of the third sentence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and according to the Top N algorithm, the fourth sentence sequence [T1, T2, ..., T k-1 T k The input is fed into the unigram language model M1 to obtain the sequence of the fourth sentence [T1, T2, ..., T]. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ].

[0022] In the above method, due to the first sentence sequence [S1, S2, ..., S...] n-1 S n The remaining fifth sentence sequence after truncation in ] k+1 S k+2 S n-1 S n ], and the second sentence sequence [T1, T2, ..., T m-1 T m The remaining seventh sentence sequence after truncation in ] ] k+1 T k+2 ,…,T m-1 T m The text data is quite long, so it can be used as a training corpus. Specifically, each corpus is converted into a corresponding text vector. Each sentence in the sentence sequence corresponds to a unigram language model. According to the Jelinek-Mercer smoothing formula, the third sentence sequence [S1, S2, ..., S...] is obtained. k-1 S k ] and the fourth sentence sequence [T1, T2, ..., T k-1 T k The probability of each language model generating the query text is calculated, i.e., the matching value between sentences is determined to match the third sentence sequence [S1, S2, ..., S]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine the sequence [T1, T2, ..., T] of the fourth sentence. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D nThis approach constructs a Jelinek-Mercer smooth optimization language model and employs linear interpolation to incorporate the word probability distribution from the corpus. This makes the language model more closely resemble the actual probability distribution, thereby improving the accuracy of long text compression results.

[0023] In yet another possible implementation, the third sentence sequence [S1, S2, ..., S...] is... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n Following that, it also includes:

[0024] The ninth sentence sequence [S1, S2, ..., S...] is used to... k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the text semantic matching model to obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] .

[0025] In the above method, after compressing the first and second initial long texts, the server can also calculate the semantic relevance or similarity between them. For example, long-text-long-text semantic matching can be used in personalized recommendation tasks by abstracting the content information of the entire long text to identify similar texts in the same domain or on the same event. Specifically, for example, the server can merge news articles (or news titles) recently viewed by a user into a long "document" and use the topic distribution of this "document" as a user profile expressing the user's reading interests. Furthermore, by calculating the Hellinger distance between the topic distribution of each real-time news article and the user profile, the similarity between the two probability distributions is obtained, which is then used as the selection criterion for pushing news to the user, thereby achieving the effect of personalized news recommendation.

[0026] In yet another possible implementation, the ninth sentence sequence [S1, S2, ..., S...] is... k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the text semantic matching model to obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] includes:

[0027] The ninth sentence sequence [S1, S2, ..., S...] is used to... k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C nThe input is fed into a dual-tower model to obtain a first output of a first tower network and a second output of a second tower network. The dual-tower model includes a first tower network and a second tower network. The first tower network is used to convert the first output into a text vector, and the second tower network of the dual-tower model is used to convert the second output into a semantic vector.

[0028] A first matching degree of the associated semantic unit encoding information is determined based on the text vector, and a second matching degree of the associated text unit encoding information is determined based on the semantic vector;

[0029] The ninth sentence sequence [S1, S2, ..., S] is determined based on the first matching degree and the second matching degree. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] .

[0030] In the above method, the server will use the ninth sentence sequence [S1, S2, ..., S...] k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the dual-tower model. Specifically, firstly, a bidirectional neural network is used in the representation layer to obtain the representation of the middle position of the text and the vector corresponding to each word position, which reflects certain global information. Then, the ninth sentence sequence [S1, S2, ..., S...] is processed. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The sentences in the sequence are interacted on a word-by-word basis to construct a matching matrix between the sentence sequences, thereby covering more detailed and local text interaction information. Convolution is used on this local matching feature matrix to extract multi-level matching features from words to an N-gram language model. Then, after convolution and a multilayer perceptron, the ninth sentence sequence [S1, S2, ..., S...] is obtained. k-1 Sk D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between two long texts. This scheme can obtain the similarity between two probability distributions by calculating the Hellinger distance between two long texts, thereby selecting the text that best matches the user input from a large database.

[0031] In yet another possible implementation, the third sentence sequence [S1, S2, ..., S...] is... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n Following that, it also includes:

[0032] Iteratively execute the following steps until the ninth sentence sequence [S1, S2, ..., S...] is reached. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n To achieve the preset target compression effect:

[0033] The ninth sentence sequence [S1, S2, ..., S...] is calculated using a machine learning algorithm. k-1 S k D1, D2, ..., D n-1 D nThe similarity between the tenth sentence sequence [T1, T2, ..., T] and the first initial long text, and the tenth sentence sequence [T1, T2, ..., T] calculated by a machine learning algorithm. k-1 T k C1, C2, ..., C n-1 C n Similarity to the second initial long text;

[0034] In response to the ninth sentence sequence [S1, S2, ..., S...] k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n If the preset target compression effect is not achieved, a new ninth sentence sequence is determined based on the first initial long text, and a new tenth sentence sequence is determined based on the second initial long text. The preset target compression effect is a state where the similarity between the new ninth sentence sequence and the first initial long text is greater than a preset threshold, and the similarity between the new tenth sentence sequence and the second initial long text is greater than a preset threshold.

[0035] In the above method, after obtaining the compressed texts of the first and second initial long texts, the server can iteratively execute the compression steps of the two initial long texts to make the compressed texts (the ninth sentence sequence [S1, S2, ..., S...)) more complete. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The ninth sentence sequence [S1, S2, ..., S...] achieves the target compression effect where the similarity to both the first and second initial long texts is greater than a preset threshold. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C nIf the preset target compression effect is not achieved, the server can determine a new ninth sentence sequence based on the first initial long text, and a new tenth sentence sequence based on the second initial long text, until the similarity between the new ninth sentence sequence and the first initial long text is greater than a preset threshold, and the similarity between the new tenth sentence sequence and the second initial long text is also greater than a preset threshold. This scheme can effectively improve the accuracy of long text compression results by promptly outputting new sentence sequences when the compressed sentence sequences do not achieve the expected compression effect, iteratively executing the step of outputting new sentence sequences until the preset target compression effect is achieved for both the first and second initial long texts.

[0036] Secondly, embodiments of this application provide a long text compression device based on a language model. The long text compression device includes a sentence segmentation unit, a truncation unit, a determination unit, and a splicing unit. The long text compression device is used to implement the method described in the first aspect or any possible implementation of the first aspect.

[0037] It should be noted that the processor included in the long text compression apparatus described in the second aspect above can be a processor specifically designed to execute these methods (referred to as a dedicated processor for distinction), or a processor that executes these methods by calling a computer program, such as a general-purpose processor. Optionally, at least one processor may include both dedicated and general-purpose processors.

[0038] Optionally, the computer program described above can be stored in memory. For example, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated with the processor on the same device or disposed on different devices. This application does not limit the type of memory or the arrangement of the memory and the processor.

[0039] In one possible implementation, at least one of the aforementioned memories is located outside the aforementioned long text compression device.

[0040] In yet another possible implementation, at least one of the aforementioned memories is located within the aforementioned long text compression device.

[0041] In another possible implementation, a portion of the memory of the at least one memory is located within the long text compression device, while another portion of the memory is located outside the long text compression device.

[0042] In this application, the processor and memory may also be integrated into a single device, that is, the processor and memory can be integrated together.

[0043] Thirdly, embodiments of this application provide a long text compression device based on a language model, the long text compression device including a processor and a memory; the memory stores a computer program; when the processor executes the computer program, the computing device executes the method described in the first aspect or any one of the first aspects.

[0044] Fourthly, embodiments of this application provide a computer-readable storage medium storing instructions that, when executed on at least one processor, implement the method described in the first aspect or any of the first aspects.

[0045] Fifthly, this application provides a computer program product comprising computer instructions that, when executed on at least one processor, implement the method described in the first aspect or any of the preceding claims. The computer program product may be a software installation package, which can be downloaded and executed on a computing device when the aforementioned method is required.

[0046] The beneficial effects of the technical methods provided in the second to fifth aspects of this application can be referred to the beneficial effects of the technical solution in the first aspect, and will not be repeated here. Attached Figure Description

[0047] The accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0048] Figure 1 This is a schematic diagram of a server architecture provided in an embodiment of this application;

[0049] Figure 2 This is a schematic diagram of a long text compression process provided in an embodiment of this application;

[0050] Figure 3 This is a flowchart illustrating a long text compression method based on a language model provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram illustrating the generation of a matching score based on the ninth sentence sequence and the tenth sentence sequence provided in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the structure of a long text compression device 50 based on a language model provided in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of the structure of a server 60 provided in an embodiment of this application. Detailed Implementation

[0054] The embodiments of this application are described below with reference to the accompanying drawings.

[0055] To facilitate understanding, the technical terms involved in the embodiments of this application will be briefly introduced first.

[0056] 1. Unigram Language Model

[0057] Statistical language models calculate the probability of a sentence. The probability of the entire sentence is the product of the probabilities of each word. A higher probability value indicates a more reasonable sentence. By processing large amounts of text corpora, the probability distribution of a given word sequence can be obtained, thus objectively describing the implicit patterns. This makes it suitable for processing large-scale real-world text. Statistical language models have been widely applied in fields such as speech recognition, machine translation, and text proofreading. N-gram is a typical statistical language model. It makes the assumption that the occurrence of the current word is only related to the preceding N-1 words and is unrelated to any other words. The probability of the entire sentence is the product of the probabilities of each word. However, it has several problems. When calculating the probability of each word, increasing N provides more prior information, making the prediction of the current word more accurate. However, when N is too large, sparsity problems occur, resulting in many words having a probability of 0. When N is 1, it is called a unigram language model.

[0058] 2. Jelinek-mercer smoothing

[0059] Jelinek-Mercer smoothing, or linear interpolation smoothing algorithm, is based on the idea of ​​linearly combining high-order and low-order models. It uses the low-order N-gram model to perform linear interpolation on the high-order N-gram model because the low-order N-gram model can often provide useful information when there is insufficient data to estimate the probability of the high-order N-gram model. By using linear interpolation, the probability distribution of words in the corpus is introduced, making the language model closer to the actual probability distribution.

[0060] 3. Jieba

[0061] Jieba is an open-source Natural Language Processing (NLP) tool that can perform NLP tasks such as part-of-speech tagging, word segmentation, and keyword extraction. It supports four segmentation modes: 1. Precise Mode: Attempts to segment the sentence as accurately as possible, outputting only the combinations with the highest probability; 2. Search Engine Mode: Based on Precise Mode, it further segments long words to improve recall, suitable for search engine segmentation; 3. Full Mode: Scans all possible words in the sentence; 4. Paddle Mode: Utilizes the PaddlePaddle deep learning framework to train a sequence labeling (bidirectional GRU) network model for word segmentation. Part-of-speech tagging is also supported.

[0062] 4. Twin Towers Model

[0063] The most significant characteristic of the dual-tower model is that "User and Item are two independent sub-networks," with the User tower on the left and the Item tower on the right. The parameters of these two towers are not shared. User features mainly include user-related features such as user ID, mobile operating system, region, age, and historical behavior sequences. Item features mainly include item-related features such as ItemId, Item category, and Item source. The User and Item features are then input into a feature extraction network (e.g., a DNN) to obtain User Embedding and Item Embedding, respectively. These two embeddings are then used as input to an interoperability layer to calculate their similarity. The cosine distance between these two embeddings is then calculated (closer to items the user has clicked, and farther to items the user hasn't clicked or dislikes). The calculated loss is then used to update the model parameters. After the model is trained, all items are processed offline through the Item tower to obtain their vectors, and these Item vectors are stored in FAISS. When a user arrives, the user vector is first calculated using the User Tower. Then, the user vector is used to calculate the similarity between the user vector and the Item vector in FAISS, and the Top K closest Items are returned as personalized recall results.

[0064] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a server provided in an embodiment of this application. The server 101 includes a data acquisition module 102, a data processing module 103, a training module 104, and an output module 105.

[0065] Server 101 can be a single server or a server cluster consisting of multiple servers, specifically a computer or a host computer. Figure 2 This is a schematic diagram of a long text compression process provided in an embodiment of this application, such as... Figure 2As shown, server 101 is mainly used to segment the first initial long text and the second initial long text into sentences and words using regular expression methods and open-source tools, respectively, to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second sentence sequence [T1, T2, ..., T] m-1 T m Then, according to the Top K algorithm, from the first sentence sequence [S1, S2, ..., S...], ... n-1 S n Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k ], and according to the Top K algorithm from the second sentence sequence [T1, T2, ..., T m-1 T m Extract the fourth sentence sequence [T1, T2, ..., T] from the data. k-1 T k Then from the fifth sentence sequence (the first sentence sequence [S1, S2, ..., S...), n-1 S n The remaining sentence sequence after truncation in ] [S k+1 S k+2 S n-1 S n Determine the sequence of the third sentence [S1, S2, ..., S] k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and from the seventh sentence sequence (the second sentence sequence [T1, T2, ..., T m-1 T m The remaining sentence sequence after truncation in [T] k+1 T k+2 ,…,T m-1 T m The sequence [T1, T2, ..., T] is determined in the context of the fourth sentence sequence. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n Finally, the third sentence sequence [S1, S2, ..., S...] is... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n ].

[0066] The data acquisition module 102 is used to acquire the first initial long text and the second initial long text.

[0067] Data processing module 103 is used to segment the first initial long text into sentences using regular expressions to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text is segmented into sentences using regular expressions to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ], and according to the Top K algorithm from the first sentence sequence [S1, S2, ..., S n-1 S n Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k ], and according to the Top K algorithm from the second sentence sequence [T1, T2, ..., T m-1 T m Extract the fourth sentence sequence [T1, T2, ..., T] from the data. k-1 T k ].

[0068] Training module 104 is used to train the remaining fifth sentence sequence after truncation [S]. k+1 S k+2 S n-1 S n Construct a unigram language model M1, and based on the remaining seventh sentence sequence [T] after truncation. k+1 T k+2 ,…,T m-1 T m Construct a unigram language model M2, and then use the Top N algorithm to sort the third sentence sequence [S1, S2, ..., S...]. k-1 S k The input is fed into the unary language model M2, resulting in the sequence of the third sentence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 Cn ], and according to the Top N algorithm, the fourth sentence sequence [T1, T2, ..., T k-1 T k The input is fed into the unigram language model M1, resulting in the sequence [T1, T2, ..., T] of the fourth sentence. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ].

[0069] In the data processing module 103, the third sentence sequence [S1, S2, ..., S...] is processed. k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n After that, the output module 105 is used to output the ninth sentence sequence [S1, S2, ..., S...] to the user. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n (i.e., the text obtained by compressing the first initial long text and the second initial long text respectively).

[0070] The methods of the embodiments of this application will be described in detail below.

[0071] Please see Figure 3 , Figure 3 This is a flowchart illustrating a long text compression method based on a language model provided in an embodiment of this application. Optionally, this method can be applied... Figure 1 The server.

[0072] like Figure 3 The language model-based long text compression method includes at least steps S301 to S304.

[0073] Step S301: The server segments the initial long text into sentences using regular expressions, obtaining the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text is segmented into sentences using the regular expression method to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ].

[0074] Optionally, in this embodiment of the application, the first initial long text and the second initial long text are two legal judgment documents that are different in content but related, and are pre-selected by the server.

[0075] Specifically, the server performs sentence segmentation on the initial long text using regular expressions as follows: First, it performs word segmentation, stop word removal, and punctuation removal on the initial long text (e.g., containing 200 sentences and 10,000 characters in total) to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text (e.g., containing 20 sentences totaling 800 characters) is processed by word segmentation, stop word removal, and punctuation removal to obtain the second sentence sequence [T1, T2, ..., T...]. m-1 T m (e.g., including 20 sentences totaling 800 characters).

[0076] Furthermore, the specific steps of the server in performing word segmentation, stop word removal, and punctuation removal preprocessing on the first and second initial long texts can be as follows: First, the first and second initial long texts are segmented into characters to obtain multiple characters corresponding to the first and second initial long texts, respectively. Then, each character in the multiple characters corresponding to the first initial long text is encoded to obtain a first feature vector, and each character in the multiple characters corresponding to the second initial long text is encoded to obtain a second feature vector. The first and second feature vectors represent the context information of each character. Then, the first sentence sequence [S1, S2, ..., S...] is determined based on the first feature vector and the language type corresponding to the first initial long text. n-1 S n ], and determine the second sentence sequence [T1, T2, ..., T] based on the second feature vector and the language type corresponding to the second initial long text. m-1 T mThis application utilizes regular expression methods and the open-source tool Jieba to process long text strings, which can effectively improve the efficiency of long text compression.

[0077] For example, if the sentences in the first initial long text end with punctuation marks of three types: ":", ",", and ".", and the content of the first initial long text is "She said: I will handle this matter, don't worry.", the server can segment the first initial long text based on these three punctuation mark types and the content of the first initial long text to obtain the first sentence sequence "She said", "I will handle this matter", and "Don't worry". Similarly, if the sentences in the second initial long text end with punctuation marks of four types: ",", ":", "、", and ".", and the content of the second initial long text is "There are many trees in this orchard, such as apple trees, pear trees, peach trees, and apricot trees", the server can segment the second initial long text based on these four punctuation mark types and the content of the second initial long text to obtain the second sentence sequence "There are many trees in this orchard", "such as", "apple trees", "pear trees", "peach trees", and "apricot trees".

[0078] Step S302: The server uses the Top K algorithm to select from the first sentence sequence [S1, S2, ..., S...] n-1 S n Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k ], and according to the Top K algorithm from the second sentence sequence [T1, T2, ..., T m-1 T m Extract the fourth sentence sequence [T1, T2, ..., T] from the data. k-1 T k ].

[0079] Specifically, the server processes the first and second initial long texts using regular expressions to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second sentence sequence [T1, T2, ..., T] m-1 T m After that, since the important information in long texts is generally concentrated at the beginning of the text, the server can use the TopK algorithm to start from the first sentence sequence [S1, S2, ..., S...]. n-1 S n Find the K largest elements from the first sentence sequence [S1, S2, ..., S...]. n-1 S n Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k(For example, if the first sentence sequence contains 20 sentences, then the first 5 consecutive sentences are extracted from the 20 sentences for subsequent operations), and the second sentence sequence [T1, T2, ..., T] is extracted according to the TopK algorithm. m-1 T m Find the K largest elements from the second sentence sequence [T1, T2, ..., T]. m-1 T m Extract the fourth sentence sequence [T1, T2, ..., T] from the data. k-1 T k (For example, if the second sentence sequence also includes 20 sentences, then the first 5 consecutive sentences are extracted from the 20 sentences for subsequent operations.)

[0080] Step S303: The server determines the sequence from the fifth sentence sequence that corresponds to the third sentence sequence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ].

[0081] The fifth sentence sequence is the first sentence sequence [S1, S2, ..., S...]. n-1 S n The remaining sentence sequence after truncation in ] [S k+1 S k+2 S n-1 S n The seventh sentence sequence is the second sentence sequence [T1, T2, ..., T]. m-1 T m The remaining sentence sequence after truncation in [T] k+1 T k+2 ,…,T m-1 T m ].

[0082] Furthermore, the server determines from the fifth sentence sequence the sequence of the third sentence [S1, S2, ..., S...] k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T kThe eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n The specific process can be as follows: The server first uses the remaining fifth sentence sequence after truncation [S] k+1 S k+2 S n-1 S n Construct a unigram language model M1, and based on the remaining seventh sentence sequence [T] after truncation. k+1 T k+2 ,…,T m-1 T m Construct a unigram language model M2; then, based on the Top N algorithm, sort the third sentence sequence [S1, S2, ..., S...]. k-1 S k The input is fed into the unary language model M2, resulting in the sequence of the third sentence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and according to the Top N algorithm, the fourth sentence sequence [T1, T2, ..., T k-1 T k The input is fed into the unigram language model M1, resulting in the sequence [T1, T2, ..., T] of the fourth sentence. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ].

[0083] Specifically, due to the first sentence sequence [S1, S2, ..., S...] n-1 S n The remaining fifth sentence sequence after truncation in ] k+1 S k+2 S n-1 S n ], and the second sentence sequence [T1, T2, ..., T m-1 T m The remaining seventh sentence sequence after truncation in ] ] k+1 T k+2 ,…,T m-1 T m The text data is quite long, so it can be used as a training corpus. Specifically, each corpus is converted into a corresponding text vector. Each sentence in the sentence sequence corresponds to a unigram language model. According to the Jelinek-Mercer smoothing formula, the third sentence sequence [S1, S2, ..., S...] is obtained. k-1 S k] and the fourth sentence sequence [T1, T2, ..., T k-1 T k The probability of each language model generating the query text is calculated, i.e., the matching value between sentences is determined to match the third sentence sequence [S1, S2, ..., S]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine the sequence [T1, T2, ..., T] of the fourth sentence. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n This approach constructs a Jelinek-Mercer smooth optimization language model and employs linear interpolation to incorporate the word probability distribution from the corpus. This makes the language model more closely resemble the actual probability distribution, thereby improving the accuracy of long text compression results.

[0084] Step S304: The server will send the third sentence sequence [S1, S2, ..., S...] k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n ].

[0085] Specifically, the server will obtain the third sentence sequence [S1, S2, ..., S...]. k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By interactively concatenating words according to their correspondence, a matching matrix is ​​constructed between the two sentence sequences. This can cover more detailed and local textual interaction information as well as semantic interaction information, making the compressed ninth sentence sequence [S1, S2, ..., S...] more comprehensive. k-1 Sk D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The accuracy rate is higher.

[0086] Optionally, the server will use the third sentence sequence [S1, S2, ..., S...] k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n After that, the ninth sentence sequence [S1, S2, ..., S...] can also be... k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the text semantic matching model to obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] .

[0087] Specifically, after compressing the first and second initial long texts, the server can also calculate the semantic relevance or similarity between them. For example, long-text-long-text semantic matching can be used in personalized recommendation tasks by abstracting the content information of the entire long text to identify similar texts in the same domain or on the same event. For instance, the server can merge recently viewed news articles (or news titles) into a long "document" and use the topic distribution of this "document" as a user profile expressing the user's reading interests. Furthermore, by calculating the Hellinger distance between the topic distribution of each real-time news article and the user profile, the similarity between the two probability distributions is obtained, which is then used as the basis for selecting news to push to the user, thereby achieving personalized news recommendation.

[0088] Furthermore, Figure 4 This is a schematic diagram illustrating the generation of matching scores based on the ninth and tenth sentence sequences provided in an embodiment of this application. Figure 4 As shown, the server will use the ninth sentence sequence [S1, S2, ..., S...] k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the text semantic matching model to obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The specific process for matching scores between the ninth sentence sequence [S1, S2, ..., S] can be as follows: k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C nThe input is fed into a dual-tower model, yielding the first output of the first tower network and the second output of the second tower network. The dual-tower model includes both a first tower network and a second tower network. The first tower network converts the first output into a text vector, and the second tower network converts the second output into a semantic vector. Then, the first matching degree of the associated semantic unit encoding information is determined based on the text vector, and the second matching degree of the associated text unit encoding information is determined based on the semantic vector. Finally, the ninth sentence sequence [S1, S2, ..., S...] is determined based on the first and second matching degrees. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] .

[0089] Specifically, the server first uses a bidirectional neural network based on the representation layer to obtain the representation of the middle position of the text and the vector corresponding to each word position, which reflects certain global information. Then, it processes the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The sentences in the sequence are interacted on a word-by-word basis to construct a matching matrix between the sentence sequences, thereby covering more detailed and local text interaction information. Convolution is used on this local matching feature matrix to extract multi-level matching features from words to an N-gram language model. Then, after convolution and a multilayer perceptron, the ninth sentence sequence [S1, S2, ..., S...] is obtained. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between two long texts. This scheme can obtain the similarity between two probability distributions by calculating the Hellinger distance between two long texts, thereby selecting the text that best matches the user input from a large database.

[0090] Optionally, after obtaining the compressed text of the first and second initial long texts, the server can iteratively perform compression steps on the two initial long texts to ensure that the compressed texts (the ninth sentence sequence [S1, S2, ..., S...)) are compressed into a single compressed text. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The ninth sentence sequence [S1, S2, ..., S...] achieves the target compression effect where the similarity to both the first and second initial long texts is greater than a preset threshold. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n If the preset target compression effect is not achieved, the server can determine a new ninth sentence sequence based on the first initial long text, and a new tenth sentence sequence based on the second initial long text, until the similarity between the new ninth sentence sequence and the first initial long text is greater than a preset threshold, and the similarity between the new tenth sentence sequence and the second initial long text is also greater than a preset threshold. This scheme can effectively improve the accuracy of long text compression results by promptly outputting new sentence sequences when the compressed sentence sequences do not achieve the expected compression effect, iteratively executing the step of outputting new sentence sequences until the preset target compression effect is achieved for both the first and second initial long texts.

[0091] In existing technologies, truncation methods can lead to insufficient model generalization or reliance on the accuracy of keyword extraction methods. Model-based methods, on the other hand, rely on complex models with too many parameters, potentially failing to meet the responsiveness requirements of real-time applications. Furthermore, the method of generating text summaries is uncontrollable and highly dependent on the generalization of the generated model. This application primarily uses a Jelinek-Mercer smoothing language model to compress long texts. Specifically, regularization methods and open-source tools are used to segment two initial long texts (comprising 200 sentences and 10,000 characters in total) into sentences and words, resulting in the first sentence sequence [S1, S2, ..., S...]. n-1 S n (Including 20 sentences totaling 800 characters) and the second sentence sequence [T1, T2, ..., T... m-1 T m[(Including 20 sentences with a total of 800 characters), then using the TopK algorithm to extract the first sentence sequence [S1, S2, ..., S...] n-1 S n The second sentence sequence [T1, T2, ..., T] m-1 T m Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k [(Including 5 sentences totaling 200 characters) and the fourth sentence sequence [T1, T2, ..., T] k-1 T k (Including 5 sentences totaling 200 characters), then based on the fifth sentence sequence remaining after extracting from the first sentence sequence [S] k+1 S k+2 S n-1 S n (Including 15 sentences totaling 600 characters) Determine the sequence [S1, S2, ..., S] of the third sentence. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and according to the sequence from the second sentence [T1, T2, ..., T m-1 T m The remaining seventh sentence sequence after truncation [T] k+1 T k+2 ,…,T m-1 T m Determine the sequence of the fourth sentence [T1, T2, ..., T] k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n Finally, the third sentence sequence [S1, S2, ..., S...] is... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n The sequence [T1, T2, ..., T] is concatenated, and the fourth sentence sequence [T1, T2, ..., T] is combined. k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n The compressed text is obtained by concatenating the words. This application is based on a Jelinek-Mercer smoothing optimized language model, which makes the probability distribution of words in the language model closer to reality, thereby improving the accuracy of long text compression results.

[0092] The methods of the embodiments of this application have been described in detail above. The apparatus of the embodiments of this application is provided below.

[0093] It is understood that the various devices provided in the embodiments of this application, such as long text compression devices, include hardware structures, software modules, or combinations of hardware structures and software structures to perform the functions described in the above method embodiments in order to achieve the functions of each device.

[0094] Those skilled in the art will readily recognize that, based on the units and algorithm steps described in conjunction with the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the foregoing method embodiments using different device implementations in different usage scenarios, and such different device implementations should not be considered beyond the scope of the embodiments of this application.

[0095] The embodiments of this application can divide the device into functional modules. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one functional module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in the embodiments of this application is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0096] For example, in the case of dividing the device into various functional modules in an integrated manner, this application provides several possible processing devices.

[0097] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a language model-based long text compression device 50 provided in an embodiment of this application. The long text compression device 50 can be... Figure 1 The server shown may be a component within that server, such as a chip, software module, or integrated circuit. The long text compression device 50 is used to implement the aforementioned long text compression method, for example... Figure 3 The long text compression method described above.

[0098] In one possible implementation, the long text compression device 50 may include a sentence segmentation unit 501, a truncation unit 502, a determination unit 503, and a splicing unit 504.

[0099] The sentence segmentation unit 501 is used to segment the first initial long text into sentences using a regular expression method to obtain a first sentence sequence [S1, S2, ..., S...]. n-1 S nThe second initial long text is segmented into sentences using the regular expression method to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ];

[0100] The truncation unit 502 is used to extract data from the first sentence sequence [S1, S2, ..., S...] using the Top K algorithm. n-1 S n Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k ], and according to the Top K algorithm from the second sentence sequence [T1, T2, ..., T m-1 T m Extract the fourth sentence sequence [T1, T2, ..., T] from the data. k-1 T k ];

[0101] The determining unit 503 is configured to determine from the fifth sentence sequence the sentence sequence [S1, S2, ..., S...] that corresponds to the third sentence sequence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ], wherein the fifth sentence sequence is the first sentence sequence [S1, S2, ..., S... n-1 S n The truncated sentence sequence [S] in k+1 S k+2 S n-1 S n The seventh sentence sequence is the second sentence sequence [T1, T2, ..., T]. m-1 T m The truncated sentence sequence [T] in k+1 T k+2 ,…,T m-1 T m ];

[0102] The splicing unit 504 is used to concatenate the third sentence sequence [S1, S2, ..., S...] k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D nBy concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n ].

[0103] In existing technologies, truncation methods can lead to insufficient model generalization or reliance on the accuracy of keyword extraction methods. Model-based methods, on the other hand, rely on complex models with too many parameters, potentially failing to meet the responsiveness requirements of real-time applications. Furthermore, the method of generating text summaries is uncontrollable and highly dependent on the generalization of the generated model. This application primarily uses the Jelinek-Mercer smoothing language model to compress long texts. Specifically, it first uses regularization methods and open-source tools to segment two initial long texts into sentences and words, obtaining the first sentence sequence [S1, S2, ..., S...]. n-1 S n (Including 20 sentences totaling 800 characters) and the second sentence sequence [T1, T2, ..., T... m-1 T m [(Including 20 sentences with a total of 800 characters), then using the TopK algorithm to extract the first sentence sequence [S1, S2, ..., S...] n-1 S n The second sentence sequence [T1, T2, ..., T] m-1 T m Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k [(Including 5 sentences totaling 200 characters) and the fourth sentence sequence [T1, T2, ..., T] k-1 T k (Including 5 sentences totaling 200 characters), then based on the fifth sentence sequence remaining after extracting from the first sentence sequence [S] k+1 S k+2 S n-1 S n (Including 15 sentences totaling 600 characters) Determine the sequence [S1, S2, ..., S] of the third sentence. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., Cn-1 C n ], and according to the sequence from the second sentence [T1, T2, ..., T m-1 T m The remaining seventh sentence sequence after truncation [T] k+1 T k+2 ,…,T m-1 T m Determine the sequence of the fourth sentence [T1, T2, ..., T] k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n Finally, the third sentence sequence [S1, S2, ..., S...] is... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n The sequence [T1, T2, ..., T] is concatenated, and the fourth sentence sequence [T1, T2, ..., T] is combined. k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n The compressed text is obtained by concatenating the words. This application is based on a Jelinek-Mercer smoothing optimized language model, which makes the probability distribution of words in the language model closer to reality, thereby improving the accuracy of long text compression results.

[0104] In another possible implementation, the first initial long text is segmented into sentences using a regular expression method to obtain a first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text is segmented into sentences using the regular expression method to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m In this regard, the sentence segmentation unit 501 is specifically used for:

[0105] The first initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the first sentence sequence [S1, S2, ..., S]. n-1 S n The second initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ].

[0106] In this embodiment, the data cleaning process using regular expressions and the open-source tool Jieba mainly involves: segmenting stop words and punctuation marks in the first and second initial long texts, preprocessing to remove stop words, and preprocessing to remove punctuation marks. Specifically, for example, if the sentence endings in the first initial long text contain three types of punctuation marks: ":", "," and ".", the server can segment the first initial long text according to these three punctuation mark types to obtain the first sentence sequence. If the first initial long text is "She said: I will handle this matter, don't worry.", then the server can segment the first initial long text into "She said", "I will handle this matter", and "Don't worry." This application utilizes regular expressions and the open-source tool Jieba to process long text strings, enabling faster and more concise matching of target string content and string processing operations such as retrieval and replacement of target objects.

[0107] In another possible implementation, after performing word segmentation, stop word removal, and punctuation removal preprocessing on the first initial long text, the first sentence sequence [S1, S2, ..., S...] is obtained. n-1 S n The second initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m In this regard, the processing unit is specifically used for:

[0108] The first initial long text is segmented into characters to obtain multiple characters corresponding to the first initial long text, and the second initial long text is segmented into characters to obtain multiple characters corresponding to the second initial long text;

[0109] Encode each character among the multiple characters corresponding to the first initial long text to obtain a first feature vector of each character among the multiple characters corresponding to the first initial long text, and encode each character among the multiple characters corresponding to the second initial long text to obtain a second feature vector of each character among the multiple characters corresponding to the second initial long text, wherein the first feature vector and the second feature vector are used to represent the context information of each character among the multiple characters;

[0110] Based on the first feature vector and the language type of the first initial long text, determine the first sentence sequence [S1, S2, ..., S...]. n-1 S n ], and based on the second feature vector and the language type of the second initial long text, determine the second sentence sequence [T1, T2, ..., T] m-1 Tm ].

[0111] In this embodiment, the specific steps of the server performing word segmentation, stop word removal, and punctuation removal preprocessing on the first and second initial long texts can be as follows: First, the two initial long texts are segmented into characters to obtain multiple characters corresponding to the initial long texts. Then, each character in the multiple characters corresponding to the initial long texts is encoded to obtain a feature vector. Finally, the first sentence sequence [S1, S2, ..., S...] is determined based on the feature vectors of the two initial long texts and the corresponding language type. n-1 S n The second sentence sequence [T1, T2, ..., T] m-1 T m (For example, the above steps are the specific process of performing regular expression processing on a first initial long text containing 200 sentences and 10,000 characters to obtain a first sentence sequence containing 20 sentences and 800 characters, and performing regular expression processing on a second initial long text containing 200 sentences and 10,000 characters to obtain a second sentence sequence containing 20 sentences and 800 characters.) This application uses regular expression methods and the open-source tool Jieba to process long text strings, which can effectively improve the efficiency of long text compression.

[0112] In another possible implementation, the third sentence sequence [S1, S2, ..., S...] is determined from the fifth sentence sequence... k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n In this regard, the determining unit 503 is specifically used for:

[0113] Based on the remaining fifth sentence sequence after truncation [S] k+1 S k+2 S n-1 S n Construct a unigram language model M1, and based on the remaining seventh sentence sequence [T] after truncation. k+1 T k+2 ,…,T m-1 T m Construct a unigram language model M2;

[0114] According to the Top N algorithm, the third sentence sequence [S1, S2, ..., S...] is used to...k-1 S k The input is fed into the unigram language model M2 to obtain the sequence of the third sentence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and according to the Top N algorithm, the fourth sentence sequence [T1, T2, ..., T k-1 T k The input is fed into the unigram language model M1 to obtain the sequence of the fourth sentence [T1, T2, ..., T]. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ].

[0115] In this embodiment of the application, due to the first sentence sequence [S1, S2, ..., S...] n-1 S n The remaining fifth sentence sequence after truncation in ] k+1 S k+2 S n-1 S n ], and the second sentence sequence [T1, T2, ..., T m-1 T m The remaining seventh sentence sequence after truncation in ] ] k+1 T k+2 ,…,T m-1 T m The text data is quite long, so it can be used as a training corpus. Specifically, each corpus is converted into a corresponding text vector. Each sentence in the sentence sequence corresponds to a unigram language model. According to the Jelinek-Mercer smoothing formula, the third sentence sequence [S1, S2, ..., S...] is obtained. k-1 S k ] and the fourth sentence sequence [T1, T2, ..., T k-1 T k The probability of each language model generating the query text is calculated, i.e., the matching value between sentences is determined to match the third sentence sequence [S1, S2, ..., S]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine the sequence [T1, T2, ..., T] of the fourth sentence. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 Dn This approach constructs a Jelinek-Mercer smooth optimization language model and employs linear interpolation to incorporate the word probability distribution from the corpus. This makes the language model more closely resemble the actual probability distribution, thereby improving the accuracy of long text compression results.

[0116] Another possible implementation also includes an input unit;

[0117] The input unit is used to process the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the text semantic matching model to obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] .

[0118] In this embodiment, after compressing the first and second initial long texts, the server can also calculate the semantic relevance or similarity between them. For example, long-text-long-text semantic matching can be used in personalized recommendation tasks by abstracting the content information of the entire long text to identify similar texts in the same domain or event. Specifically, for example, the server can merge recently viewed news (or news titles) into a long "document" and use the topic distribution of this "document" as a user profile expressing the user's reading interests. Furthermore, by calculating the Hellinger distance between the topic distribution of each real-time news article and the user profile, the similarity between the two probability distributions is obtained, which is then used as the selection criterion for pushing news to the user, thereby achieving the effect of personalized news recommendation.

[0119] In another possible implementation, when the ninth sentence sequence [S1, S2, ..., S...] is... k-1 S k D1, D2, ..., D n-1 D n] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the text semantic matching model to obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n Regarding the matching score between [ ], the input unit is specifically used for:

[0120] The ninth sentence sequence [S1, S2, ..., S...] is used to... k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into a dual-tower model to obtain a first output of a first tower network and a second output of a second tower network. The dual-tower model includes a first tower network and a second tower network. The first tower network is used to convert the first output into a text vector, and the second tower network of the dual-tower model is used to convert the second output into a semantic vector.

[0121] A first matching degree of the associated semantic unit encoding information is determined based on the text vector, and a second matching degree of the associated text unit encoding information is determined based on the semantic vector;

[0122] The ninth sentence sequence [S1, S2, ..., S] is determined based on the first matching degree and the second matching degree. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] .

[0123] In this embodiment of the application, the server will use the ninth sentence sequence [S1, S2, ..., S...] k-1 S kD1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the dual-tower model. Specifically, firstly, a bidirectional neural network is used in the representation layer to obtain the representation of the middle position of the text and the vector corresponding to each word position, which reflects certain global information. Then, the ninth sentence sequence [S1, S2, ..., S...] is processed. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The sentences in the sequence are interacted on a word-by-word basis to construct a matching matrix between the sentence sequences, thereby covering more detailed and local text interaction information. Convolution is used on this local matching feature matrix to extract multi-level matching features from words to an N-gram language model. Then, after convolution and a multilayer perceptron, the ninth sentence sequence [S1, S2, ..., S...] is obtained. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between two long texts. This scheme can obtain the similarity between two probability distributions by calculating the Hellinger distance between two long texts, thereby selecting the text that best matches the user input from a large database.

[0124] Another possible implementation also includes a computing unit;

[0125] Iteratively execute the following steps until the ninth sentence sequence [S1, S2, ..., S...] is reached. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n To achieve the preset target compression effect:

[0126] The computing unit is used to calculate the ninth sentence sequence [S1, S2, ..., S] using a machine learning algorithm. k-1 S k D1, D2, ..., D n-1 D n The similarity between the tenth sentence sequence [T1, T2, ..., T] and the first initial long text, and the tenth sentence sequence [T1, T2, ..., T] calculated by a machine learning algorithm. k-1 T k C1, C2, ..., C n-1 C n Similarity to the second initial long text;

[0127] The determining unit 503 is further configured to respond to the ninth sentence sequence [S1, S2, ..., S...] k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n If the preset target compression effect is not achieved, a new ninth sentence sequence is determined based on the first initial long text, and a new tenth sentence sequence is determined based on the second initial long text. The preset target compression effect is a state where the similarity between the new ninth sentence sequence and the first initial long text is greater than a preset threshold, and the similarity between the new tenth sentence sequence and the second initial long text is greater than a preset threshold.

[0128] In this embodiment of the application, after obtaining the compressed text of the first initial long text and the second initial long text, the server can iteratively execute the compression steps of the two initial long texts so that the two compressed texts (the ninth sentence sequence [S1, S2, ..., S...)) are compressed into a single compressed text. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The ninth sentence sequence [S1, S2, ..., S...] achieves the target compression effect where the similarity to both the first and second initial long texts is greater than a preset threshold. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1T k C1, C2, ..., C n-1 C n If the preset target compression effect is not achieved, the server can determine a new ninth sentence sequence based on the first initial long text, and a new tenth sentence sequence based on the second initial long text, until the similarity between the new ninth sentence sequence and the first initial long text is greater than a preset threshold, and the similarity between the new tenth sentence sequence and the second initial long text is also greater than a preset threshold. This scheme can effectively improve the accuracy of long text compression results by promptly outputting new sentence sequences when the compressed sentence sequences do not achieve the expected compression effect, iteratively executing the step of outputting new sentence sequences until the preset target compression effect is achieved for both the first and second initial long texts.

[0129] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a server 60 provided in an embodiment of this application, including components such as chips, software modules, and integrated circuits. The server 60 may include at least one processor 601. Optionally, it may also include at least one memory 603. Further optionally, the server 60 may also include a communication interface 602. Even more optionally, it may also include a bus 604, wherein the processor 601, the communication interface 602, and the memory 603 are connected via the bus 604.

[0130] The processor 601 is a module that performs arithmetic and / or logical operations. Specifically, it can be one or a combination of processing modules such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), a coprocessor (to assist the central processing unit in completing corresponding processing and applications), and a microcontroller unit (MCU).

[0131] The communication interface 602 can be used to provide information input or output to the at least one processor. And / or, the communication interface 602 can be used to receive data transmitted externally and / or transmit data externally, and can be a wired link interface including an Ethernet cable, or a wireless link interface (Wi-Fi, Bluetooth, general wireless transmission, vehicular short-range communication technology, and other short-range wireless communication technologies, etc.). Optionally, the communication interface 602 may also include a transmitter (such as a radio frequency transmitter, antenna, etc.) or a receiver coupled to the interface.

[0132] The memory 603 provides storage space, in which data such as the operating system and computer programs can be stored. The memory 603 can be one or a combination of several of the following: Random Access Memory (RAM), Read-Only Memory (ROM), Erasable Programmable Read-Only Memory (EPROM), or Compact Disc Read-Only Memory (CD-ROM).

[0133] At least one processor 601 in the server 60 is used to perform the aforementioned method, for example Figure 3 The method described in the embodiments.

[0134] Optionally, processor 601 can be a processor specifically designed to execute these methods (for clarity, referred to as a dedicated processor), or a processor that executes these methods by invoking a computer program, such as a general-purpose processor. Optionally, at least one processor may include both dedicated and general-purpose processors. Optionally, when the computing device includes at least one processor 601, the aforementioned computer program may be stored in memory 603.

[0135] Optionally, at least one processor 601 in the server 60 is configured to execute computer instructions to perform the following operations:

[0136] The initial long text is segmented into sentences using regular expressions to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text is segmented into sentences using the regular expression method to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ], wherein there is a correlation between the first initial long text and the second initial long text;

[0137] According to the Top K algorithm, from the first sentence sequence [S1, S2, ..., S...] n-1 S n Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k ], and according to the Top K algorithm from the second sentence sequence [T1, T2, ..., T m-1 T m Extract the fourth sentence sequence [T1, T2, ..., T] from the data. k-1 T k ];

[0138] Determine the sequence from the fifth sentence sequence that corresponds to the third sentence sequence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ], wherein the fifth sentence sequence is the first sentence sequence [S1, S2, ..., S... n-1 S n The remaining sentence sequence after truncation in ] [S k+1 S k+2 S n-1 S n The seventh sentence sequence is the second sentence sequence [T1, T2, ..., T]. m-1 T m The remaining sentence sequence after truncation in [T] k+1 T k+2 ,…,T m-1 T m ];

[0139] The third sentence sequence [S1, S2, ..., S...] is used to... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n ].

[0140] In existing technologies, truncation methods can lead to insufficient model generalization or reliance on the accuracy of keyword extraction methods. Model-based methods, on the other hand, rely on complex models with too many parameters, potentially failing to meet the responsiveness requirements of real-time applications. Furthermore, the method of generating text summaries is uncontrollable and highly dependent on the generalization of the generated model. This application primarily uses the Jelinek-Mercer smoothing language model to compress long texts. Specifically, it first uses regularization methods and open-source tools to segment two initial long texts into sentences and words, obtaining the first sentence sequence [S1, S2, ..., S...]. n-1 S n (Including 20 sentences totaling 800 characters) and the second sentence sequence [T1, T2, ..., T... m-1 T m [(Including 20 sentences with a total of 800 characters), then using the TopK algorithm to extract the first sentence sequence [S1, S2, ..., S...] n-1 S n The second sentence sequence [T1, T2, ..., T] m-1 T m Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k [(Including 5 sentences totaling 200 characters) and the fourth sentence sequence [T1, T2, ..., T] k-1 T k (Including 5 sentences totaling 200 characters), then based on the fifth sentence sequence remaining after extracting from the first sentence sequence [S] k+1 S k+2 S n-1 S n (Including 15 sentences totaling 600 characters) Determine the sequence [S1, S2, ..., S] of the third sentence. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and according to the sequence from the second sentence [T1, T2, ..., T m-1 T m The remaining seventh sentence sequence after truncation [T] k+1 T k+2 ,…,T m-1 Tm Determine the sequence of the fourth sentence [T1, T2, ..., T] k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n Finally, the third sentence sequence [S1, S2, ..., S...] is... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n The sequence [T1, T2, ..., T] is concatenated, and the fourth sentence sequence [T1, T2, ..., T] is combined. k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n The compressed text is obtained by concatenating the words. This application is based on a Jelinek-Mercer smoothing optimized language model, which makes the probability distribution of words in the language model closer to reality, thereby improving the accuracy of long text compression results.

[0141] Optionally, the processor 601 is further configured to:

[0142] The first initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the first sentence sequence [S1, S2, ..., S]. n-1 S n The second initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ].

[0143] In this embodiment, the data cleaning process using regular expressions and the open-source tool Jieba mainly involves: segmenting stop words and punctuation marks in the first and second initial long texts, preprocessing to remove stop words, and preprocessing to remove punctuation marks. Specifically, for example, if the sentence endings in the first initial long text contain three types of punctuation marks: ":", "," and ".", the server can segment the first initial long text according to these three punctuation mark types to obtain the first sentence sequence. If the first initial long text is "She said: I will handle this matter, don't worry.", then the server can segment the first initial long text into "She said", "I will handle this matter", and "Don't worry." This application utilizes regular expressions and the open-source tool Jieba to process long text strings, enabling faster and more concise matching of target string content and string processing operations such as retrieval and replacement of target objects.

[0144] Optionally, the processor 601 is further configured to:

[0145] The first initial long text is segmented into characters to obtain multiple characters corresponding to the first initial long text, and the second initial long text is segmented into characters to obtain multiple characters corresponding to the second initial long text;

[0146] Encode each character among the multiple characters corresponding to the first initial long text to obtain a first feature vector of each character among the multiple characters corresponding to the first initial long text, and encode each character among the multiple characters corresponding to the second initial long text to obtain a second feature vector of each character among the multiple characters corresponding to the second initial long text, wherein the first feature vector and the second feature vector are used to represent the context information of each character among the multiple characters;

[0147] Based on the first feature vector and the language type of the first initial long text, determine the first sentence sequence [S1, S2, ..., S...]. n-1 S n ], and based on the second feature vector and the language type of the second initial long text, determine the second sentence sequence [T1, T2, ..., T] m-1 T m ].

[0148] In this embodiment, the specific steps of the server performing word segmentation, stop word removal, and punctuation removal preprocessing on the first and second initial long texts can be as follows: First, the two initial long texts are segmented into characters to obtain multiple characters corresponding to the initial long texts. Then, each character in the multiple characters corresponding to the initial long texts is encoded to obtain a feature vector. Finally, the first sentence sequence [S1, S2, ..., S...] is determined based on the feature vectors of the two initial long texts and the corresponding language type. n-1 S n The second sentence sequence [T1, T2, ..., T] m-1 T m (For example, the above steps are the specific process of performing regular expression processing on a first initial long text containing 200 sentences and 10,000 characters to obtain a first sentence sequence containing 20 sentences and 800 characters, and performing regular expression processing on a second initial long text containing 200 sentences and 10,000 characters to obtain a second sentence sequence containing 20 sentences and 800 characters.) This application uses regular expression methods and the open-source tool Jieba to process long text strings, which can effectively improve the efficiency of long text compression.

[0149] Optionally, the processor 601 is further configured to:

[0150] Based on the remaining fifth sentence sequence after truncation [S] k+1 S k+2 S n-1 S n Construct a unigram language model M1, and based on the remaining seventh sentence sequence [T] after truncation. k+1 T k+2 ,…,T m-1 T m Construct a unigram language model M2;

[0151] According to the Top N algorithm, the third sentence sequence [S1, S2, ..., S...] is used to... k-1 S k The input is fed into the unigram language model M2 to obtain the sequence of the third sentence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and according to the Top N algorithm, the fourth sentence sequence [T1, T2, ..., T k-1 T k The input is fed into the unigram language model M1 to obtain the sequence of the fourth sentence [T1, T2, ..., T]. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ].

[0152] In this embodiment of the application, due to the first sentence sequence [S1, S2, ..., S...] n-1 S n The remaining fifth sentence sequence after truncation in ] k+1 S k+2 S n-1 S n ], and the second sentence sequence [T1, T2, ..., T m-1 T m The remaining seventh sentence sequence after truncation in ] ] k+1 T k+2 ,…,T m-1 T m The text data is quite long, so it can be used as a training corpus. Specifically, each corpus is converted into a corresponding text vector. Each sentence in the sentence sequence corresponds to a unigram language model. According to the Jelinek-Mercer smoothing formula, the third sentence sequence [S1, S2, ..., S...] is obtained. k-1 S k ] and the fourth sentence sequence [T1, T2, ..., Tk-1 T k The probability of each language model generating the query text is calculated, i.e., the matching value between sentences is determined to match the third sentence sequence [S1, S2, ..., S]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine the sequence [T1, T2, ..., T] of the fourth sentence. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n This approach constructs a Jelinek-Mercer smooth optimization language model and employs linear interpolation to incorporate the word probability distribution from the corpus. This makes the language model more closely resemble the actual probability distribution, thereby improving the accuracy of long text compression results.

[0153] Optionally, the processor 601 is further configured to:

[0154] The ninth sentence sequence [S1, S2, ..., S...] is used to... k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the text semantic matching model to obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] .

[0155] In this embodiment, after compressing the first and second initial long texts, the server can also calculate the semantic relevance or similarity between them. For example, long-text-long-text semantic matching can be used in personalized recommendation tasks by abstracting the content information of the entire long text to identify similar texts in the same domain or event. Specifically, for example, the server can merge recently viewed news (or news titles) into a long "document" and use the topic distribution of this "document" as a user profile expressing the user's reading interests. Furthermore, by calculating the Hellinger distance between the topic distribution of each real-time news article and the user profile, the similarity between the two probability distributions is obtained, which is then used as the selection criterion for pushing news to the user, thereby achieving the effect of personalized news recommendation.

[0156] Optionally, the processor 601 is further configured to:

[0157] The ninth sentence sequence [S1, S2, ..., S...] is used to... k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into a dual-tower model to obtain a first output of the first tower network and a second output of the second tower network. The dual-tower model includes the first tower network and the second tower network. The first tower network is used to convert the first output into a text vector, and the second tower network of the dual-tower model is used to convert the second output into a semantic vector.

[0158] A first matching degree of the associated semantic unit encoding information is determined based on the text vector, and a second matching degree of the associated text unit encoding information is determined based on the semantic vector;

[0159] The ninth sentence sequence [S1, S2, ..., S] is determined based on the first matching degree and the second matching degree. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] .

[0160] In this embodiment of the application, the server will use the ninth sentence sequence [S1, S2, ..., S...] k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the dual-tower model. Specifically, firstly, a bidirectional neural network is used in the representation layer to obtain the representation of the middle position of the text and the vector corresponding to each word position, which reflects certain global information. Then, the ninth sentence sequence [S1, S2, ..., S...] is processed. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The sentences in the sequence are interacted on a word-by-word basis to construct a matching matrix between the sentence sequences, thereby covering more detailed and local text interaction information. Convolution is used on this local matching feature matrix to extract multi-level matching features from words to an N-gram language model. Then, after convolution and a multilayer perceptron, the ninth sentence sequence [S1, S2, ..., S...] is obtained. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between two long texts. This scheme can obtain the similarity between two probability distributions by calculating the Hellinger distance between two long texts, thereby selecting the text that best matches the user input from a large database.

[0161] Optionally, the processor 601 is further configured to:

[0162] Iteratively execute the following steps until the ninth sentence sequence [S1, S2, ..., S...] is reached. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C nTo achieve the preset target compression effect:

[0163] The ninth sentence sequence [S1, S2, ..., S...] is calculated using a machine learning algorithm. k-1 S k D1, D2, ..., D n-1 D n The similarity between the tenth sentence sequence [T1, T2, ..., T] and the first initial long text, and the tenth sentence sequence [T1, T2, ..., T] calculated by a machine learning algorithm. k-1 T k C1, C2, ..., C n-1 C n Similarity to the second initial long text;

[0164] In response to the ninth sentence sequence [S1, S2, ..., S...] k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n If the preset target compression effect is not achieved, a new ninth sentence sequence is determined based on the first initial long text, and a new tenth sentence sequence is determined based on the second initial long text. The preset target compression effect is a state where the similarity between the new ninth sentence sequence and the first initial long text is greater than a preset threshold, and the similarity between the new tenth sentence sequence and the second initial long text is greater than a preset threshold.

[0165] In this embodiment of the application, after obtaining the compressed text of the first initial long text and the second initial long text, the server can iteratively execute the compression steps of the two initial long texts so that the two compressed texts (the ninth sentence sequence [S1, S2, ..., S...)) are compressed into a single compressed text. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The ninth sentence sequence [S1, S2, ..., S...] achieves the target compression effect where the similarity to both the first and second initial long texts is greater than a preset threshold. k-1 S k D1, D2, ..., D n-1 D n] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n If the preset target compression effect is not achieved, the server can determine a new ninth sentence sequence based on the first initial long text, and a new tenth sentence sequence based on the second initial long text, until the similarity between the new ninth sentence sequence and the first initial long text is greater than a preset threshold, and the similarity between the new tenth sentence sequence and the second initial long text is also greater than a preset threshold. This scheme can effectively improve the accuracy of long text compression results by promptly outputting new sentence sequences when the compressed sentence sequences do not achieve the expected compression effect, iteratively executing the step of outputting new sentence sequences until the preset target compression effect is achieved for both the first and second initial long texts.

[0166] This application also provides a computer-readable storage medium storing instructions that, when executed on at least one processor, implement the aforementioned language model-based long text compression method, for example... Figure 3 The method described.

[0167] This application also provides a computer program product comprising computer instructions that, when executed by a computing device, implement the aforementioned language model-based long text compression method, for example... Figure 3 The method described.

[0168] In this application, the terms "for example" or "for instance" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "for example" or "for instance" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the use of terms such as "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0169] In this application, "at least one" in the embodiments refers to one or more items, and "more than one" refers to two or more items. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent: a, b, c, (a and b), (a and c), (b and c), or (a and b and c), where a, b, and c can be single or multiple. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0170] Furthermore, unless otherwise stated, the use of ordinal numbers such as "first" and "second" in the embodiments of this application is for distinguishing multiple objects and is not for limiting the order, sequence, priority, or importance of multiple objects. For example, "first device" and "second device" are only for ease of description and do not indicate that the first device and the second device are different in structure, importance, etc. In some embodiments, the first device and the second device may also be the same device.

[0171] In the above embodiments, the term "when..." can be interpreted, depending on the context, as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". The above descriptions are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the concept and principles of this application should be included within the protection scope of this application.

[0172] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0173] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A long text compression method based on a language model, characterized in that, The method includes: The initial long text is segmented into sentences using regular expressions to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text is segmented into sentences using the regular expression method to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ], wherein there is a correlation between the first initial long text and the second initial long text; According to the Top K algorithm, from the first sentence sequence [S1, S2, ..., S...] n-1 S n Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k ], and according to the Top K algorithm from the second sentence sequence [T1, T2, ..., T m-1 T m Extract the fourth sentence sequence [T1, T2, ..., T] from the data. k-1 T k ]; Determine the sequence from the fifth sentence sequence that corresponds to the third sentence sequence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ], wherein the fifth sentence sequence is the first sentence sequence [S1, S2, ..., S... n-1 S n The remaining sentence sequence after truncation in ] [S k+1 S k+2 S n-1 S n The seventh sentence sequence is the second sentence sequence [T1, T2, ..., T]. m-1 T m The remaining sentence sequence after truncation in [T] k+1 T k+2 ,…,T m-1 T m ]; The third sentence sequence [S1, S2, ..., S...] is used to... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n ]; The step of determining from the fifth sentence sequence the relationship with the third sentence sequence [S1, S2, ..., S...] k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ],include: Based on the remaining fifth sentence sequence after truncation [S] k+1 S k+2 S n-1 S n Construct a unigram language model M1, and based on the remaining seventh sentence sequence [T] after truncation. k+1 T k+2 ,…,T m-1 T m Construct a unigram language model M2; According to the Top N algorithm, the third sentence sequence [S1, S2, ..., S...] is used to... k-1 S k The input is fed into the unigram language model M2 to obtain the sequence of the third sentence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and according to the Top N algorithm, the fourth sentence sequence [T1, T2, ..., T k-1 T k The input is fed into the unigram language model M1 to obtain the sequence of the fourth sentence [T1, T2, ..., T]. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ].

2. The method according to claim 1, characterized in that, The first initial long text is segmented into sentences using a regular expression method to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text is segmented into sentences using the regular expression method to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ],include: The first initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the first sentence sequence [S1, S2, ..., S]. n-1 S n The second initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ].

3. The method according to claim 2, characterized in that, The first initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text is processed by word segmentation, stop word removal, and punctuation removal to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ],include: The first initial long text is segmented into characters to obtain multiple characters corresponding to the first initial long text, and the second initial long text is segmented into characters to obtain multiple characters corresponding to the second initial long text; Encode each character among the multiple characters corresponding to the first initial long text to obtain a first feature vector of each character among the multiple characters corresponding to the first initial long text, and encode each character among the multiple characters corresponding to the second initial long text to obtain a second feature vector of each character among the multiple characters corresponding to the second initial long text, wherein the first feature vector and the second feature vector are used to represent the context information of each character among the multiple characters; Based on the first feature vector and the language type of the first initial long text, determine the first sentence sequence [S1, S2, ..., S...]. n-1 S n ], and based on the second feature vector and the language type of the second initial long text, determine the second sentence sequence [T1, T2, ..., T] m-1 T m ].

4. The method according to claim 1, characterized in that, The third sentence sequence [S1, S2, ..., S...] is... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n Following that, it also includes: The ninth sentence sequence [S1, S2, ..., S...] is used to... k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the text semantic matching model to obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] .

5. The method according to claim 4, characterized in that, The ninth sentence sequence [S1, S2, ..., S...] is... k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into the text semantic matching model to obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] includes: The ninth sentence sequence [S1, S2, ..., S...] is used to... k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The input is fed into a dual-tower model to obtain a first output of a first tower network and a second output of a second tower network. The dual-tower model includes a first tower network and a second tower network. The first tower network is used to convert the first output into a text vector, and the second tower network of the dual-tower model is used to convert the second output into a semantic vector. A first matching degree of the associated semantic unit encoding information is determined based on the text vector, and a second matching degree of the associated text unit encoding information is determined based on the semantic vector; The ninth sentence sequence [S1, S2, ..., S] is determined based on the first matching degree and the second matching degree. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n The matching score between ] .

6. The method according to claim 1, characterized in that, The third sentence sequence [S1, S2, ..., S...] is... k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n Following that, it also includes: Iteratively execute the following steps until the ninth sentence sequence [S1, S2, ..., S...] is reached. k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n To achieve the preset target compression effect: The ninth sentence sequence [S1, S2, ..., S...] is calculated using a machine learning algorithm. k-1 S k D1, D2, ..., D n-1 D n The similarity between the tenth sentence sequence [T1, T2, ..., T] and the first initial long text, and the tenth sentence sequence [T1, T2, ..., T] calculated by a machine learning algorithm. k-1 T k C1, C2, ..., C n-1 C n Similarity to the second initial long text; In response to the ninth sentence sequence [S1, S2, ..., S...] k-1 S k D1, D2, ..., D n-1 D n ] and the tenth sentence sequence [T1, T2, ..., T k-1 T k C1, C2, ..., C n-1 C n If the preset target compression effect is not achieved, a new ninth sentence sequence is determined based on the first initial long text, and a new tenth sentence sequence is determined based on the second initial long text. The preset target compression effect is a state where the similarity between the new ninth sentence sequence and the first initial long text is greater than a preset threshold, and the similarity between the new tenth sentence sequence and the second initial long text is greater than a preset threshold.

7. A long text compression device based on a language model, characterized in that, It includes sentence segmentation units, extraction units, determination units, and splicing units, among which: The sentence segmentation unit is used to segment the first initial long text into sentences using a regular expression method to obtain the first sentence sequence [S1, S2, ..., S...]. n-1 S n The second initial long text is segmented into sentences using the regular expression method to obtain the second sentence sequence [T1, T2, ..., T]. m-1 T m ]; The truncation unit is used to extract data from the first sentence sequence [S1, S2, ..., S...] using the Top K algorithm. n-1 S n Extract the third sentence sequence [S1, S2, ..., S] from the data. k-1 S k ], and according to the Top K algorithm from the second sentence sequence [T1, T2, ..., T m-1 T m Extract the fourth sentence sequence [T1, T2, ..., T] from the data. k-1 T k ]; The determining unit is used to determine from the fifth sentence sequence the sentence sequence [S1, S2, ..., S...] that corresponds to the third sentence sequence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ], wherein the fifth sentence sequence is the first sentence sequence [S1, S2, ..., S... n-1 S n The truncated sentence sequence [S] in k+1 S k+2 S n-1 S n The seventh sentence sequence is the second sentence sequence [T1, T2, ..., T]. m-1 T m The truncated sentence sequence [T] in k+1 T k+2 ,…,T m-1 T m ]; The splicing unit is used to combine the third sentence sequence [S1, S2, ..., S...] k-1 S k ] and the eighth sentence sequence [D1, D2, ..., D n-1 D n By concatenating these sequences, we obtain the ninth sentence sequence [S1, S2, ..., S...]. k-1 S k D1, D2, ..., D n-1 D n ], and the fourth sentence sequence [T1, T2, ..., T k-1 T k ] and the sixth sentence sequence [C1, C2, ..., C n-1 C n By concatenating these sequences, we obtain the tenth sentence sequence [T1, T2, ..., T]. k-1 T k C1, C2, ..., C n-1 C n ]; The fifth sentence sequence is determined to be related to the third sentence sequence [S1, S2, ..., S]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and determine from the seventh sentence sequence the sequence of the fourth sentence [T1, T2, ..., T k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n In this regard, the determining unit is specifically used for: Based on the remaining fifth sentence sequence after truncation [S] k+1 S k+2 S n-1 S n Construct a unigram language model M1, and based on the remaining seventh sentence sequence [T] after truncation. k+1 T k+2 ,…,T m-1 T m Construct a unigram language model M2; According to the Top N algorithm, the third sentence sequence [S1, S2, ..., S...] is used to... k-1 S k The input is fed into the unigram language model M2 to obtain the sequence of the third sentence [S1, S2, ..., S...]. k-1 S k The sixth sentence sequence with the highest similarity [C1, C2, ..., C n-1 C n ], and according to the Top N algorithm, the fourth sentence sequence [T1, T2, ..., T k-1 T k The input is fed into the unigram language model M1 to obtain the sequence of the fourth sentence [T1, T2, ..., T]. k-1 T k The eighth sentence sequence with the highest similarity [D1, D2, ..., D...] n-1 D n ].

8. A server, characterized in that, The server includes a processor and a memory, the memory being used to store computer instructions, and the processor being used to invoke the computer instructions to implement the method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on at least one processor, implement the method as described in any one of claims 1-6.

Images