Large Model-based Long Text Matching Method, System, Terminal and Medium

Abstract

The present invention provides a long text matching method, system, terminal and medium based on a large model, including: extracting first key semantic information of a first long text by using a first large model; calculating a matching similarity with a second long text based on the first key semantic information; wherein, calculating the matching similarity with the second long text based on the first key semantic information further includes any one of the following: dividing the first key semantic information into corresponding keywords, and calculating the matching similarity between the keywords and the second long text; extracting second key semantic information of the second long text by using the first large model; calculating the matching similarity between the first key semantic information and the second key semantic information. The present invention helps long texts to be dimensionally reduced through a large model, first solves the problem of text processing, can better retain the key information in the original long text, and at the same time solves the problem of high video memory occupation of traditional methods.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing. Specifically, it relates to a long text matching method and system based on a large model, as well as a corresponding computer terminal and computer-readable storage medium. Background Art

[0002] Calculating the similarity of long texts in this field remains a difficult problem. In actual business scenarios, long texts are not necessarily well-processed and unambiguous simple texts (i.e., the data used in academia, which can be directly used for training). Actual long texts may contain a lot of noise, such as various symbols with unclear meanings, spaces, line breaks, etc. First, the obtained data needs to be cleaned to obtain clean pure text information, and then the similarity of long texts is calculated. Currently, the methods for calculating the similarity of long texts are roughly divided into two categories. One is the deep learning method. Currently, the better deep learning models for calculating the similarity of long texts are the Sentence-BERT and Longformer models. However, for such deep learning models, it is very difficult to handle the similarity between long texts of thousands or tens of thousands of words because when the text is very long, the training dataset has extremely high requirements for the device's video memory. Another method is the SimHash algorithm. The main work of the SimHash algorithm is to reduce the dimension of the text, generate a SimHash value, and then compare the Hamming distance through the SimHash values of different texts to determine the similarity of the two texts. The characteristic is that when the text content is long, the accuracy of using SimHash is relatively high, but the accuracy of SimHash for processing short text content often cannot be guaranteed. Using such methods for calculation is relatively complex and cannot understand semantics.

[0003] After retrieval, it is found that the Chinese patent application for invention "A Text Matching Method for Long Texts" with the publication number CN117828028A uses a dual-tower Longformer model to build the text information of the text pairs to be matched. As described above, the Longformer model used in this method has high requirements for video memory and is very difficult to handle texts of tens of thousands of words. Summary of the Invention

[0004] In view of the above deficiencies in the prior art, the present invention provides a long text matching method, system, terminal and medium based on a large model.

[0005] According to one aspect of the present invention, a long text matching method based on a large model is provided, including:

[0006] Using a first large model to extract the first key semantic information of a first long text;

[0007] Based on the first key semantic information, performing a matching similarity calculation with a second long text;

[0008] Among them, the matching similarity calculation between the first key semantic information and the second long text further includes any one of the following:

[0009] - Divide the first key semantic information into corresponding keywords, and calculate the matching similarity between the keywords and the second long text;

[0010] - Use the first large model to extract the second key semantic information of the second long text; calculate the matching similarity between the first key semantic information and the second key semantic information.

[0011] According to the second aspect of the present invention, a long text matching system based on a large model is provided, including:

[0012] A first key information extraction module, which uses the first large model to extract the first key semantic information of the first long text;

[0013] A similarity matching module, which calculates the matching similarity with the second long text based on the first key semantic information;

[0014] Among them, the similarity matching module further includes any one of the following:

[0015] - A direct matching unit, which is used to divide the first key semantic information into corresponding keywords, and calculate the matching similarity between the keywords and the second long text;

[0016] - A second key information extraction unit and a similarity calculation unit; where:

[0017] The second key information extraction unit uses the first large model to extract the second key semantic information of the second long text;

[0018] The similarity calculation unit is used to calculate the matching similarity between the first key semantic information and the second key semantic information.

[0019] According to the third aspect of the present invention, another long text matching method based on a large model is provided, including:

[0020] Use the second large model to perform dimensionality reduction processing on several known long text pairs with marked matching degrees to construct a training data set;

[0021] Use the training data set to train a deep learning model;

[0022] Take the new long text pair as the input of the trained deep learning model, and output the matching degree score.

[0023] According to the fourth aspect of the present invention, another long text matching system based on a large model is provided, including:

[0024] A training sample construction module, which uses a second large model to perform dimensionality reduction processing on a number of known long text pairs with marked matching degrees to construct a training data set;

[0025] A model training module, which uses the training data set to train a deep learning model;

[0026] A matching module, which is used to take a new long text pair as the input of the trained deep learning model and output a matching degree score.

[0027] According to the fifth aspect of the present invention, a computer terminal is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it can be used to execute the method described above in the present invention, or run the system described above in the present invention.

[0028] According to the sixth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it can be used to execute the method described above in the present invention, or run the system described above in the present invention.

[0029] Due to the adoption of the above technical solutions, compared with the prior art, the present invention has at least one of the following beneficial effects:

[0030] The present invention uses a large model to generate keywords, solving the problem that it is difficult for traditional models to perform secondary generation on long texts.

[0031] The present invention matches the keywords of the first long text with the second long text, solving the situation that traditional models can only perform classification matching according to preset topics. These keywords are closer to the original text information than the preset topics, avoiding the problem of low matching degree in the follow-up.

[0032] The present invention calculates the matching similarity between the first key semantic information and the second key semantic information, solving the text matching situation where the text contents are different but the semantics are the same.

[0033] The present invention adopts the technology of processing long texts based on a large model, solving the problem that general long text matching technologies require high device video memory. On the basis of retaining semantic information matching, the accuracy is ensured.

[0034] The present invention uses a large model to help reduce the dimensionality of long texts, first solving the problem of text processing, being able to better retain the key information in the original long texts, and at the same time solving the problem of high video memory occupation by traditional methods. On the basis of retaining semantic information matching, the accuracy is ensured. Brief Description of the Drawings

[0035] Other features, objects, and advantages of the present invention will become more apparent by reading the following detailed description of non - restrictive embodiments with reference to the accompanying drawings:

[0036] Figure 1 It is a flowchart of the first long - text matching method based on a large model in an embodiment of the present invention.

[0037] Figure 2 It is a flowchart of the first matching similarity calculation in a preferred embodiment of the present invention.

[0038] Figure 3 It is a flowchart of the second matching similarity calculation in a preferred embodiment of the present invention.

[0039] Figure 4 It is a schematic diagram of the component modules of the first long - text matching system based on a large model in an embodiment of the present invention.

[0040] Figure 5 It is a flowchart of the second long - text matching method based on a large model in an embodiment of the present invention.

[0041] Figure 6 It is a schematic diagram of the component modules of the second long - text matching system based on a large model in an embodiment of the present invention.

[0042] Figure 7 It is a flowchart of the second long - text matching system based on a large model in a preferred embodiment of the present invention. Detailed Embodiment

[0043] The following is a detailed description of the embodiments of the present invention: These embodiments are implemented on the premise of the technical solution of the present invention, and detailed implementation manners and specific operation processes are given. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can still be made, and these all belong to the protection scope of the present invention.

[0044] Existing long - text matching technologies have high requirements for video memory, are difficult to process texts with tens of thousands of characters, have complex calculations, and cannot understand semantics. In view of the above problems, an embodiment of the present invention provides a long - text matching method based on a large model. This method uses a large model to help reduce the dimension of long texts, first solves the problem of text processing, can better retain the key information in the original long text, and at the same time solves the problem of high video - memory occupation of traditional methods, and ensures accuracy while retaining semantic information matching.

[0045] Specifically, as Figure 1As shown, the long text matching method based on a large model provided by this embodiment may include:

[0046] S1. Extract the first key semantic information of the first long text using the first large model;

[0047] S2. Calculate the matching similarity with the second long text based on the first key semantic information;

[0048] Among them, calculating the matching similarity with the second long text based on the first key semantic information may further include any one or any combination of the following:

[0049] S21. Divide the first key semantic information into corresponding keywords, and calculate the matching similarity between the keywords and the second long text;

[0050] S22. Extract the second key semantic information of the second long text using the first large model; Calculate the matching similarity between the first key semantic information and the second key semantic information.

[0051] In some preferred embodiments, in S11 above, the first large model may adopt the following pre-trained large models: any non-encoder-only model. Further, any large model with inductive generation ability can be used as the first large model. This large model can be trained through a self-constructed dataset or other pre-trained open-source large models. <s

[0052] In some preferred embodiments, as Figure 2 shown, in S21 above, dividing the first key semantic information into corresponding keywords and calculating the matching similarity between the keywords and the second long text may further include:

[0053] S211. Divide the key semantic information generated by the first large model into corresponding keywords. Further preferably:

[0054] By setting corresponding prompt words, even including using the Chain of Thought (CoT) technology that enhances the reasoning ability of large language models, the required keyword information can be generated. In addition, according to grammar rules, regular expressions can be optionally used for secondary splitting to obtain the desired keywords.

[0055] S222. Compare the generated keywords with the second long text, and the number of matching keywords is the similarity between the two texts. Further preferably, it includes any one of the following methods:

[0056] - Identify and count the frequency of the generated keywords (i.e., the number of keywords) in the second-longest text, and assign corresponding weight scores according to the importance of the keywords. The weighted score of the number of keywords is the similarity between the two texts.

[0057] - A vocabulary containing synonyms, near synonyms, or specific abbreviations of keywords can be pre-constructed. When these words appear in the second-longest text, it is considered a successful match, and the number of matched words is the similarity between the two texts.

[0058] In S21 above, the more keywords are matched, the higher the matching degree between the two texts. The matching similarity calculation method provided in this step is applicable to very long texts with a lot of professional information. By extracting these key semantic information for comparison, the relevance between two long texts can be effectively judged. For example, when both texts are academic papers, the method provided in this step has better effect.

[0059] In some preferred embodiments, as Figure 3 shown, in S22 above, use the first large model to extract the second key semantic information of the second-longest text; calculate the matching similarity between the first key semantic information and the second key semantic information, and may further include:

[0060] S221, pass the second-longest text through the first large model to extract key semantic information, and obtain the second key semantic information;

[0061] S222, calculate the similarity between the first key semantic information and the second key semantic information, including: Pearson correlation coefficient, Euclidean distance, Cosine similarity, and Manhattan distance calculation methods.

[0062] In S22 above, calculate the similarity between these two key semantic information to determine whether two long texts are relevant. The matching similarity calculation method provided in this step is more applicable to general texts with relatively weak professionalism.

[0063] Based on the same inventive concept, an embodiment of the present invention provides a long text matching system based on a large model.

[0064] Specifically, as Figure 4 shown, the long text matching system based on a large model provided in this embodiment may include:

[0065] The first key information extraction module, which uses the first large model to extract the first key semantic information of the first-longest text;

[0066] The similarity matching module, which calculates the matching similarity with the second-longest text based on the first key semantic information;

[0067] Among them, the similarity matching module may further include any one or any combination of the following:

[0068] - A direct matching unit, which is used to divide the first key semantic information into corresponding keywords and calculate the similarity between the keywords and the second long text;

[0069] - A second key information extraction unit and a similarity calculation unit; where:

[0070] The second key information extraction unit uses the first large model to extract the second key semantic information of the second long text;

[0071] The similarity calculation unit is used to calculate the similarity between the first key semantic information and the second key semantic information.

[0072] Next, the working content of each functional module of the long text matching system based on the large model provided in the above embodiments of the present invention will be further described in detail.

[0073] In this system, the obtained long text is summarized by the large model to obtain the key semantic information. At present, most mainstream large models have good capabilities in extracting key semantic information from long texts. Of course, for specific business scenarios, the model with corresponding fine-tuning will have better effects. The key semantic information obtained by summarization generally already includes the key semantics of the long text. Next, the similarity calculation with another long text can be implemented through any functional unit of the first key information extraction module and the similarity matching module.

[0074] Using the direct matching unit; first, the key semantic information generated by the first large model needs to be divided into corresponding keywords through the first key information extraction module, and then the generated keywords are compared with another long text. The number of matching keywords is the similarity between the two texts. The more matching keywords, the higher the matching degree between the two texts. This method is applicable to two very long texts that contain a lot of professional information. Extracting these key information for comparison can effectively judge the relevance between the two texts. For example, when both texts are academic papers, this method has good effects.

[0075] Adopt a second key information extraction unit and a similarity calculation unit; first extract the first key semantic information of a long text passed through the first large model through the first key information extraction module, and extract the second key semantic information of another long text passed through the first large model through the second key information extraction unit; then calculate the similarity of these two key semantic information through the similarity calculation unit. The similarity calculation includes methods such as Pearson correlation coefficient, Euclidean distance, Cosine similarity, Manhattan distance, etc. These calculation methods are relatively common methods in machine learning. This method is suitable for judging whether two texts are relevant. Because these methods require that the text cannot be too long, the information extracted after the large model can be understood as the titles of these two texts. This method is more suitable for ordinary texts with relatively weak professionalism.

[0076] An embodiment of the present invention also provides another long text matching method based on a large model.

[0077] Specifically, as Figure 5 shown, the long text matching method based on a large model provided by this embodiment may include:

[0078] M1, use the second large model to perform dimensionality reduction processing on several known long text pairs with marked matching degrees, and construct a training data set;

[0079] M2, use the training data set to train a deep learning model;

[0080] M3, take the new long text pair as the input of the trained deep learning model, and output the matching degree score.

[0081] In some preferred embodiments, for M1 above, the second large model can adopt the following pre-trained model: any non-encoder-only model.

[0082] In some preferred embodiments. For M1 above, using the second large model to perform dimensionality reduction processing on several known long text pairs with marked matching degrees and construct a training data set can further include:

[0083] M11, use the second large model to extract the key information of several known long text pairs with marked matching degrees, perform dimensionality reduction on the known long text, and form new texts;

[0084] M12, combine the new texts with the marked matching degrees to form a training data set.

[0085] Based on the same inventive concept, an embodiment of the present invention also provides another long text matching system based on a large model.

[0086] Specifically, as Figure 6As shown, the long text matching system based on a large model provided by this embodiment may include:

[0087] A training sample construction module, which uses a second large model to perform dimensionality reduction on a number of known long text pairs with marked matching degrees, and constructs a training data set;

[0088] A model training module, which uses the training data set to train a deep learning model;

[0089] A matching module, which is used to take a new long text pair as the input of the trained deep learning model and output a matching degree score.

[0090] In some preferred embodiments, the above training sample construction module may further include:

[0091] A long text dimensionality reduction unit, which uses the second large model to extract key information of a number of known long text pairs with marked matching degrees, performs dimensionality reduction on the known long texts, and forms new texts;

[0092] A data set construction unit, which is used to combine the new texts with the marked matching degrees to form a training data set.

[0093] The working contents of each functional module constituting the long text matching system based on a large model provided by the above embodiment of the present invention will be further described in detail below.

[0094] In this system, by constructing a training data set, training a deep learning model is used to obtain the matching degree score of unknown long text pairs, so as to judge the similarity between long texts and achieve long text matching.

[0095] As Figure 7 shown, further, obtain a number of known long text pairs, and the matching degree of the text pairs has been marked for the known long text pairs. According to the video memory of the existing device, use a pre-trained large model to perform data dimensionality reduction on the known long text pairs. After experimental testing, when training a text pair of about 5000 words using the Longformer model, the occupied video memory is about 800m. According to the existing device, reduce the length of the text data to the range available to the device for training the deep learning model. Finally, the trained deep learning model can be used to judge the similarity between texts. Here, the large model mainly plays a role in removing the noise of the original text, retaining useful information, reducing the text length, and reducing the requirements for the device. The deep learning model can adopt the Transformer model, or can be the SentenceBert model and the Longformer model based on the Transformer architecture.

[0096] An embodiment of the present invention further provides a computer terminal, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it can be used to execute the method of any one of the above embodiments of the present invention, or to run the system of any one of the above embodiments of the present invention.

[0097] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random access memory (English: random-access memory, abbreviation: RAM), such as static random access memory (English: static random-access memory, abbreviation: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviation: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as application programs and functional modules for implementing the above methods), computer instructions, etc. The above computer programs, computer instructions, etc. can be stored in one or more memories in partitions. And the above computer programs, computer instructions, data, etc. can be called by the processor.

[0098] The processor is used to execute the computer program stored in the memory to implement each step in the method or each module in the system involved in the above embodiments. For specific details, reference can be made to the relevant descriptions in the previous method and system embodiments.

[0099] The processor and the memory can be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor can be coupled and connected through a bus.

[0100] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it can be used to execute the method of any one of the above embodiments of the present invention, or to run the system of any one of the above embodiments of the present invention.

[0101] Among them, the computer-readable medium includes computer storage media and communication media, where the communication media includes any medium facilitating the transmission of computer programs from one place to another. The storage media can be any available medium accessible by a general or special-purpose computer. An exemplary storage media is coupled to the processor, enabling the processor to read information from and write information to the storage media. Of course, the storage media can also be part of the processor. The processor and the storage media can be located in an ASIC. Additionally, the ASIC can be located in a user device. Of course, the processor and the storage media can also exist as discrete components in a communication device.

[0102] The long text matching method, system, terminal, and medium based on a large model provided in the above embodiments of the present invention adopt the technology of processing long texts based on a large model, solving the problem that general long text matching technologies require high device video memory. On the basis of retaining semantic information matching, the accuracy is ensured; by helping long texts to reduce dimensions through the large model, the problem of text processing is first solved, the key information in the original long text can be better retained, and at the same time, the problem of high video memory occupation by traditional methods is solved. On the basis of retaining semantic information matching, the accuracy is ensured.

[0103] Matters not described in detail in the above embodiments of the present invention are all well-known technologies in the art.

[0104] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various deformations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A long text matching method based on a large model, characterized in that: include: Extracting first key semantic information of the first long text using the first large model; Based on the first key semantic information, performing matching similarity calculation with the second long text; Also includes: Divide the first key semantic information into corresponding keywords, and perform matching similarity calculation between the keywords and the second long text; wherein: The dividing the first key semantic information into corresponding keywords includes: By using the preset prompt words and / or thought chain technology, the key semantic information generated by the first model is divided into corresponding first keywords; based on grammatical rules, the first keywords are split twice using regular expressions to generate the final desired keywords; The matching similarity calculation between the keyword and the second long text includes any one of the following: Compare the generated keywords with the second long text and count the number of matching keywords; assign weight scores based on the importance of the corresponding keywords, and calculate the weighted score of the number of keywords, which is the similarity between the two texts; or The generated keywords are used to construct a vocabulary containing keywords, keyword synonyms, keyword antonyms and / or specific abbreviations related to the keywords; the vocabulary is compared with the second long text, and the number of matched words is the similarity between the two texts.

2. A long text matching system based on a large model, characterized by: include: A first key information extraction module, which uses the first large model to extract first key semantic information of the first long text; A similarity matching module, which performs matching similarity calculation with the second long text based on the first key semantic information; wherein the similarity matching module further includes: A direct matching unit, which is used to divide the first key semantic information into corresponding keywords and perform matching similarity calculation between the keywords and the second long text; wherein, in the direct matching unit: The dividing the first key semantic information into corresponding keywords includes: By using the preset prompt words and / or thought chain technology, the key semantic information generated by the first model is divided into corresponding first keywords; based on grammatical rules, the first keywords are split twice using regular expressions to generate the final desired keywords; The matching similarity calculation between the keyword and the second long text includes any one of the following: Compare the generated keywords with the second long text and count the number of matching keywords; assign weight scores based on the importance of the corresponding keywords, and calculate the weighted score of the number of keywords, which is the similarity between the two texts; or The generated keywords are used to construct a vocabulary containing keywords, keyword synonyms, keyword antonyms and / or specific abbreviations related to the keywords; the vocabulary is compared with the second long text, and the number of matched words is the similarity between the two texts.

3. A computer terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When executing the computer program, the processor can be used to perform the method according to claim 1 or run the system according to claim 2.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it can be used to perform the method according to claim 1 or run the system according to claim 2.

Images