Text abstract generation method based on large language model distillation framework

Abstract

The invention discloses a text abstract generation method based on a large language model distillation framework. The method comprises the following steps: acquiring a plurality of target training samples; any one target training sample is processed through an encoder of the abstract abstract model, and features of any one target training sample are obtained; and processing the features of any target training sample and any target training sample through a basic aspect module, an associated sentence module and a triple entity relationship module to obtain initial basic aspect features, initial associated sentence features and initial triple entity relationship features. The technical problems that in the prior art, abstracts are extracted from texts through LLMs, and the generated abstracts are low in quality, controllability, interpretability and stability are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a text summary generation method based on a large language model distillation framework. Background Art

[0002] Text summarization (TS) aims to automatically compress long texts (such as articles, reports or book chapters) into short and coherent summaries while retaining the main information and core ideas of the original text. This process requires that the summary can accurately convey the essence of the original text while reducing the length of the text, ensuring that readers can quickly obtain key information without having to read the full text. Text summarization technology has a wide range of application values ​​in many fields such as information retrieval, news reporting, and academic research. Although text summarization technology has made significant progress in recent years, especially driven by deep learning-based models, there are still some challenges: 1) Balance between information retention and compression: How to effectively compress the length of the text while retaining the key information of the original text and avoiding the loss of important information is a major problem facing text summarization technology. 2) Coherence and readability of the summary: The generated summary not only needs to contain the core information of the original text, but also needs to maintain linguistic coherence and good readability, which is crucial to improving user experience. 3) Domain adaptability: Texts in different fields have different characteristics and language styles. How to make the text summary model have cross-domain adaptability is a hot topic in current research. 4) Distinguishing between abstraction and extraction: A summary can be a direct extraction of the original text (extractive summary), or it can be a new text generated by understanding and reorganizing the original text (generative summary). How to find the best balance between these two methods is also a focus of current research.

[0003] Recently, the research on Large Language Models (LLMs) has achieved remarkable results and swept the vast field of Natural Language Processing (NLP). Its excellent performance and extraordinary text creativity in many tasks are remarkable. These models can not only deeply understand the subtleties of human language, but also easily master the task of natural language generation under the conditions of zero or very few samples, opening up an unprecedented exploration path for researchers. The researchers cleverly integrated the powerful text generation capabilities of LLMs into the summary generation process, aiming to automatically extract the essence from lengthy texts and quickly generate concise and accurate content summaries.

[0004] However, despite the broad prospects of this direction, the following problems still exist in practice: 1) LLMs may be too faithful to the original text when generating summaries, resulting in lengthy summaries and lack of necessary refinement and generalization. 2) In pursuit of conciseness, important details of the original text are ignored, resulting in information loss. 3) The adaptability of the model to texts of different fields and styles needs to be improved. 4) The speed and quality of summary generation of LLMs still need to be improved, and they are not as efficient as small language models (SLMs). How to ensure that the generated summaries are in line with the reading habits of the target audience and can accurately convey the original intention has become an urgent problem to be solved. In response to the above problems, researchers are actively exploring various strategies to optimize the summary generation method based on LLMs. For example, by introducing additional constraints or reward mechanisms, the model is guided to pay more attention to the selection and generalization of information when generating summaries; at the same time, the model's learning and understanding of domain knowledge is strengthened to improve its accuracy and efficiency when processing texts in specific fields. In addition, reinforcement learning technology combined with human feedback also provides new ideas for further improving the quality of summaries. Summary of the invention

[0005] The embodiment of the present invention provides a text summary generation method based on a large language model distillation framework, so as to at least solve the technical problems in the prior art of extracting summaries from texts through LLMs, resulting in low quality, controllability, interpretability and stability of the generated summaries.

[0006] According to one aspect of an embodiment of the present invention, a method for generating text summaries based on a large language model distillation framework is provided. The method may include: obtaining multiple target training samples; processing any target training sample through an encoder of an abstract summary model to obtain features of any target training sample; processing any target training sample and features of any target training sample through a basic aspect module, an associated sentence module, and a triple entity relationship module to obtain initial basic aspect features, initial associated sentence features, and initial triple entity relationship features; calculating cosines of the initial basic aspect features, initial associated sentence features, and initial triple entity relationship features with the target basic aspect features, target associated sentence features, and target triple entity relationship features extracted by the structured principle of any target training sample, respectively. loss, so that the cosine loss is minimized, wherein the structural principle of any target training sample is obtained by processing any target training sample through a large language model; the initial basic aspect features, initial associated sentence features and initial triple entity relationship features are input into their respective entry neural networks to obtain features corresponding to the initial basic aspect features, features corresponding to the initial associated sentence features and features corresponding to the initial triple entity relationship features; the features corresponding to the initial basic aspect features, the features corresponding to the initial associated sentence features and the features corresponding to the initial triple entity relationship features are fused with features of different levels of the decoder of the abstract summary model to output a target summary of any target training sample.

[0007] Optionally, before obtaining multiple target training samples, the method also includes: obtaining multiple initial training samples, wherein each initial training sample includes text and a standard summary; using a large language model, combining each sub-structuring principle and the text portion of an initial training sample corresponding to each sub-structuring principle for processing to obtain multiple sub-summaries, wherein each sub-structuring principle corresponds to a sub-summary; compressing the multiple sub-summaries through the large language model to obtain a total summary of each initial training sample; obtaining a quality score of the structured principle of each initial training sample based on the total summary and the standard summary of each initial training sample; and screening the multiple initial training samples based on the quality score of the structured principle of each initial training sample to obtain multiple target training samples.

[0008] Optionally, the expression for calculating the cosine loss by respectively comparing the initial basic aspect features, the initial associated sentence features and the initial triple entity relationship features with the target basic aspect features, the target associated sentence features and the target triple entity relationship features extracted by the structured principle of any target training sample is:

[0009] L EA =sim out , >

[0010] L AS =yes out , <s>>

[0011] L TER =yes <R out , <r>>

[0012] Among them, L EA is the cosine loss between the initial basic aspect feature and the target basic aspect feature, A out is the initial basic aspect feature,< / r> < / s> ​ <s> The basic characteristics of the target; L AS is the cosine loss between the initial associated sentence feature and the target associated sentence feature, S out is the initial associated sentence feature, <s>is the target associated sentence feature; L TER is the cosine loss between the initial triple entity relationship features and the target triple entity relationship features, R out is the initial triple entity relationship feature, <r>is the target triple entity relationship feature.

[0013] Optionally, the method further includes: outputting an expression of a composite loss function of a target summary of any target training sample as follows:

[0014]

[0015] Among them, L is the composite loss function, is the loss function of the abstract summary model, and γ1 and γ2 are two hyperparameters.

[0016] Optionally, before processing the text portion of each initial training sample through each sub-structuring principle of the large language model to obtain multiple sub-abstracts, the method also includes: extracting multiple subject words corresponding to each standard summary sentence in each initial training sample; extracting multiple related sentences of the multiple subject words corresponding to each standard summary sentence in each initial training sample; extracting triple entity relationships from each related sentence to obtain the triple entity relationship of each standard summary sentence; based on multiple subject words, multiple related sentences, and the triple entity relationship of each standard summary sentence, obtaining the structuring principle of the standard summary of each initial training sample; wherein the triple entity relationship of a subject word, a related sentence, and a standard summary sentence is a sub-structuring principle.

[0017] Optionally, the text portion of each initial training sample is processed by each substructuring principle of the large language model to obtain the expression of multiple sub-summaries:

[0018]

[0019] in, is the sub-summary corresponding to the ith sentence, ~ is the probability distribution, S mul are multiple sub-summaries, D is each training sample, R is the triple entity relationship of each training sample, |R| represents the number of structured principles in the triple entity relationship of each training sample, and r i * represents the triple entity relationship corresponding to the i-th sentence, Represents the sub-summary corresponding to the first i sentences.

[0020] Optionally, the expression of compressing multiple sub-summaries by using a large language model to obtain a total summary of each initial training sample is:

[0021]

[0022] in, is the total summary corresponding to the i-th sentence, S tol is the total summary for each initial training sample, Represents the total summary corresponding to the first i sentences.

[0023] Optionally, based on the total summary and the standard summary of each initial training sample, obtaining a quality score of the structured principle of each initial training sample includes:

[0024] When each training sample is a long document, the quality score of the structured principle of each initial training sample is expressed as:

[0025]

[0026] Among them, n is used to represent the number of phrases in each sub-abstract of the total abstract of each initial training sample. The value range of n is 1 to 2. match (gram n ) indicates the number of phrases that match the words in the general abstract and each sub-abstract of the standard abstract. Count(gram n ) is the number of phrases n in each sub-abstract of the standard summary, and Score1 represents the quality score of the structured principle when the initial training sample is a long document.

[0027] When each training sample is a short document, the quality score of the structured principle of each initial training sample is expressed as:

[0028]

[0029] The value range of n is 1 to 3, and Score2 represents the quality score of the structured principle when the initial training sample is a long document.

[0030] Beneficial effects of the present invention:

[0031] The present invention proposes a text summarization method based on a large language model distillation framework. Based on the structured principle-guided subframework (LSR) of LLM, three subtasks are designed for LLM to "mine" the EA, AS and TER of the document. At the same time, these elements constitute the structured rationale, providing insights into the document from three different perspectives. Secondly, in order to ensure the overall quality of the structured reasons, a multi-step summary generation evaluation strategy is adopted to select high-quality structured reasons for subsequent local small model training. Finally, a multi-view weak gating fusion subframework (MWF) is used to train the local small model. Specifically, three hierarchical modules are added to gradually align the functions of EA, AS and TER. These features are then controlled by a weak gating mechanism to fuse the output of the summary model, solving the technical problems of low quality, controllability, interpretability and stability of the summary generated by extracting and summarizing text through LLMs in the prior art, achieving the goal of converting the text summary generation task into a three-stage task, effectively utilizing the outstanding structured principle reasoning ability of LLM, and hierarchically distilling the structured principles to SLM, improving the quality of its summary generation, as well as the technical effects of controllability, interpretability and stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0033] < / r> < / s> <s> Figure 1 is a flowchart of a method for generating a text summary based on a large language model distillation framework according to an embodiment of the present invention;

[0034] Figure 2 It is an architectural diagram of a text summary generation method based on a large language model distillation framework according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Example 1

[0038] According to an embodiment of the present invention, a method for generating a text summary based on a large language model distillation framework is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system comprising at least one set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0039] Figure 1 is a flowchart of a method for generating a text summary based on a large language model distillation framework according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0040] Step S101, obtaining multiple target training samples.

[0041] In the technical solution provided in the above step S101 of the present invention, a plurality of target training samples are obtained, each of which includes a text and a standard summary part. Figure 2 is an architecture diagram of a text summary generation method based on a large language model distillation framework according to an embodiment of the present invention, Figure 2 The documents in step 3 in are multiple target training samples.

[0042] Step S102: Process any target training sample through the encoder of the abstract summary model to obtain the features of any target training sample.

[0043] In the technical solution provided in step S102 of the present invention, if Figure 2 As shown, the encoder of the abstract summary model in step three processes any target training sample to obtain the features of any target training sample.

[0044] Step S103, processing any target training sample and any target training sample's features through the basic aspect module, the associated sentence module and the triple entity relationship module to obtain initial basic aspect features, initial associated sentence features and initial triple entity relationship features.

[0045] In the technical solution provided in step S103 of the present invention, Figure 2 As shown, according to the EA module in step three, any target training sample is processed, the features of any target training sample are fused with the cross attention in the EA module, and the fused features are input into the cross attention in the AS module for feature fusion. The self-attention in the AS module processes any target training sample to obtain the self-attention features in the AS module, and the fused features in the AS module are input into the cross attention in the TER module. The self-attention in the TER module processes any target training sample to obtain the self-attention features in the TER module, and the initial basic aspect features (EA features), initial associated sentence features (AS features) and initial triple entity relationship features (TER features) are obtained.

[0046] Step S104, calculate the cosine loss by comparing the initial basic aspect features, initial associated sentence features and initial triple entity relationship features with the target basic aspect features, target associated sentence features and target triple entity relationship features extracted by the structured principle of any target training sample, respectively, to minimize the cosine loss, wherein the structured principle of any target training sample is obtained by processing any target training sample with a large language model.

[0047] In the technical solution provided in step S104 of the present invention, if Figure 2 As shown, the cosine similarity of the target basic aspect features, target associated sentence features and target triple entity relationship features of any target training sample extracted by the optimal structured principle is calculated respectively to obtain the cosine loss, and the network is adjusted to minimize the cosine loss.

[0048] Step S105, input the initial basic aspect features, initial associated sentence features and initial triple entity relationship features into respective entry neural networks to obtain features corresponding to the initial basic aspect features, features corresponding to the initial associated sentence features and features corresponding to the initial triple entity relationship features.

[0049] In the technical solution provided in step S105 of the present invention, if Figure 2 As shown, the initial basic aspect features, initial associated sentence features, and initial triple entity relationship features are input into their respective weak gates to obtain features corresponding to the initial basic aspect features, features corresponding to the initial associated sentence features, and features corresponding to the initial triple entity relationship features.

[0050] Step S105, the features corresponding to the initial basic aspect features, the features corresponding to the initial associated sentence features and the features corresponding to the initial triple entity relationship features are merged with the features of different levels of the decoder of the abstract summary model to output a target summary of any target training sample.

[0051] In the technical solution provided in step S105 of the present invention, if Figure 2 As shown in Figure 1, the degree of fusion of various features extracted from different perspectives by each decoding layer is dynamically adjusted according to the current context. out , S out and R out After entering their respective weakly gated networks, each is replicated L times, where L represents the number of layers in the decoder of the summary model. The output X of the encoder in the summary model is obtained en Finally, in the decoding stage, the following operations are performed at each layer of the decoder.

[0052]

[0053] Among them, L refers to the number of network layers of the encoder, l∈L, which means a number between l=1,2,3,...,12, which means the lth network layer. On the left side of the formula, X represents the feature, de represents that this feature is the feature of the decoder (that is, the decoder), and l+1 represents the l+1th network layer. On the right side of the formula, MulHead(Q,K,V) represents a network structure in the encoder and decoder, which is called the "multi-head attention mechanism" module. The input is Q, K and V", which corresponds to the three "WX" from left to right in the formula, for example is "Q", where "W l Q ” represents the network weight of the “Q” attribute of the lth layer, and represents the characteristics of the decoder at layer l, Represents the result of hierarchical feature fusion between the output of the summary model encoder (encoder, abbreviated as en) and the structured principle.

[0054] here, is the result of hierarchical feature fusion between the output of the summary model encoder and the structured principle:

[0055]

[0056] in, represents the weak gating unit of layer l, A out,l S out,l R out,l Represents the EA feature, AS feature and TER feature of the lth network layer, which are specially designed to be combined with Related semantic features. The value range is [0, 1]. It is a continuous value that can be adaptively updated during training. and Function and Similar. Different from previous work, which uses a fixed ReLU activation function as a gate for non-updatable weights, the gate is regarded as a network parameter for adaptive learning weights, called a weak gating unit. By observing the values ​​of weak gating units in different layers, we can understand the model's dependence on different features at different decoding stages, which helps to better understand the decision-making process and working mechanism of the summary model.

[0057] The above method of this embodiment is further introduced below.

[0058] As an optional implementation manner, step S101, before obtaining multiple target training samples, the method also includes: obtaining multiple initial training samples, wherein each initial training sample includes text and a standard summary; using a large language model, combining each sub-structuring principle and the text portion of an initial training sample corresponding to each sub-structuring principle for processing to obtain multiple sub-summaries, wherein each sub-structuring principle corresponds to a sub-summary; compressing the multiple sub-summaries through the large language model to obtain a total summary of each initial training sample; based on the total summary and the standard summary of each initial training sample, obtaining a quality score of the structured principle of each initial training sample; and screening the multiple initial training samples based on the quality score of the structured principle of each initial training sample to obtain multiple target training samples.

[0059] In this embodiment, if Figure 2 As shown, step two, the best structuring principle is selected, and multiple initial training samples are obtained, wherein each initial training sample includes text and a standard summary; through a large language model, each sub-structuring principle and the text part of an initial training sample corresponding to each sub-structuring principle are processed to obtain multiple sub-summaries, wherein each sub-structuring principle corresponds to a sub-summary; the multiple sub-summaries are compressed through the large language model to obtain a total summary of each initial training sample; according to the total summary and the standard summary of each initial training sample, the quality score of the structuring principle of each initial training sample is obtained; according to the quality score of the structuring principle of each initial training sample, the multiple initial training samples are screened to obtain multiple target training samples.

[0060] As an optional implementation method, in step S104, the initial basic aspect features, initial associated sentence features and initial triple entity relationship features are respectively compared with the target basic aspect features, target associated sentence features and target triple entity relationship features extracted by the structured principle of any target training sample, and the expression for calculating the cosine loss is:

[0061] L EA =sim out , >

[0062] L AS =yes out , <s>>

[0063] L TER =yes <R out , <r>>

[0064] Among them, L EA is the cosine loss between the initial basic aspect feature and the target basic aspect feature, A out is the initial basic aspect feature,< / r> < / s> ​ <s> The basic characteristics of the target; L AS is the cosine loss between the initial associated sentence feature and the target associated sentence feature, S out is the initial associated sentence feature, <s>is the target associated sentence feature; L TER is the cosine loss between the initial triple entity relationship features and the target triple entity relationship features, R out is the initial triple entity relationship feature, <r>is the target triple entity relationship feature.

[0065] In this embodiment, the cosine loss between the initial basic aspect features and the target basic aspect features, the cosine distance between the initial associated sentence features and the target associated sentence features, and the cosine distance between the initial triple entity relationship features and the target triple entity relationship features are calculated by the above formula.

[0066] As an optional embodiment, in step S106, the method further includes: outputting an expression of a composite loss function of a target summary of any target training sample as follows:

[0067]

[0068] Among them, L is the composite loss function, is the loss function of the abstract summary model, and γ1 and γ2 are two hyperparameters.

[0069] In this embodiment, if< / r> < / s> <s> Figure 2 As shown, the composite loss function in step 3 is the product of the loss function of the abstract summary model and γ1 and L EA , L AS , L TER The sum of the product of the sum and γ2.

[0070] As an optional implementation method, before processing the text portion of each initial training sample through each sub-structuring principle of a large language model to obtain multiple sub-abstracts, the method also includes: extracting multiple subject words corresponding to each standard summary sentence in each initial training sample; extracting multiple related sentences of the multiple subject words corresponding to each standard summary sentence in each initial training sample; extracting triple entity relationships from each related sentence to obtain the triple entity relationship of each standard summary sentence; based on the triple entity relationship of multiple subject words, multiple related sentences, and each standard summary sentence, obtaining the structuring principle of the standard summary of each initial training sample; wherein the triple entity relationship of a subject word, a related sentence, and a standard summary sentence is a sub-structuring principle.

[0071] In this embodiment, a sub-framework (LSR) guided by the structural principle of LLM, using the powerful extraction and generation capabilities of LLM and tailored prompt templates, conducts extensive exploration of the potential reasoning process between the source document and its corresponding standard summary, called "structural principle". This exploration is guided by the needs of the subsequent three tasks through hierarchical mining of three different perspectives: Essential Aspects (EA), Associated Sentence (AS), and Triple Entity Relations (TER). In order to facilitate further detailed discussion, some important concepts are outlined below.

[0072] Essential Aspect (EA): defined as the subject term a corresponding to each summary sentence extracted from the source document 1～l .

[0073] Associated Sentence (AS): defined as the most relevant sentence s in the document corresponding to the EA 1～n .

[0074] Ternary Entity Relationship (TER): defined as a structured ternary entity relationship (m≥n) extracted from AS, including object s 1～m , the relationship 1～m and object o 1～m .

[0075] Task 1 (EA extraction): This task is defined as extracting the EA in document D and its corresponding standard summary S to finally obtain the set A = {a1, a2, ..., a l }. Let |S * | indicates S * The number of sentences in , then the formula is:

[0076]

[0077] Among them, a λ It means extracting keywords corresponding to each standard summary sentence from document D. represents the i-th standard summary sentence, a <i Indicates the keywords corresponding to the first i sentences.

[0078] Task 2 (AS extraction): Given the extracted EA, this task is defined as extracting AS from document D to form a set S = {s1, s2, ..., s n },in:

[0079]

[0080] s i It means extracting the data related to a from document D. λ The corresponding sentence, s <i Indicates the associated sentences corresponding to the first i keywords.

[0081] Task 3 (TER extraction): Based on the extracted AS, this task is defined as i Extract the corresponding TER from in:

[0082]

[0083] in, Indicates that each s i Extract the corresponding TER, r *,<1 Indicates the TER corresponding to the first i key sentences.

[0084] As an optional embodiment, the text portion of each initial training sample is processed by each sub-structuring principle of the large language model to obtain the expression of multiple sub-summaries:

[0085]

[0086] in, is the sub-summary corresponding to the ith sentence, ~ is the probability distribution, S mul are multiple sub-summaries, D is each training sample, R is the triple entity relationship of each training sample, |R| represents the number of structured principles in the triple entity relationship of each training sample, and r i * represents the triple entity relationship corresponding to the i-th sentence, Represents the sub-summary corresponding to the first i sentences.

[0087] As an optional implementation manner, the expression of the total summary of each initial training sample obtained by compressing multiple sub-summaries through a large language model is:

[0088]

[0089] in, is the total summary corresponding to the i-th sentence, S tol is the total summary for each initial training sample, Represents the total summary corresponding to the first i sentences.

[0090] As an optional embodiment, based on the total summary and the standard summary of each initial training sample, obtaining the quality score of the structured principle of each initial training sample includes: when each training sample is a long document, the expression of the quality score of the structured principle of each initial training sample is:

[0091]

[0092] Among them, n is used to represent the number of phrases in each sub-abstract of the total abstract of each initial training sample. The value range of n is 1 to 2. match (gram n ) indicates the number of phrases that match the words in the general abstract and each sub-abstract of the standard abstract. Count(gram n ) is the number of phrases n in each sub-abstract of the standard summary, and Score1 represents the quality score of the structured principle when the initial training sample is a long document.

[0093] When each training sample is a short document, the quality score of the structured principle of each initial training sample is expressed as:

[0094]

[0095] The value range of n is 1 to 3, and Score2 represents the quality score of the structured principle when the initial training sample is a long document.

[0096] In this embodiment, when the value range of n is 1 to 2, indicating that each training sample is a long document, the quality score of the structuring principle of the long document is calculated by the above formula; when the value range of n is 1 to 3, indicating that each training sample is a short document, the quality score of the structuring principle of the short document is calculated by the above formula.

[0097] Comparison of experimental results:

[0098] Table 1 Performance comparison of the baseline model of the present invention on CNNDM and XSum datasets

[0099]

[0100] Table 2 Performance comparison of different modules of the present invention on CNNDM and XSum datasets

[0101]

[0102] In an embodiment of the present invention, a plurality of target training samples are obtained; any target training sample is processed by an encoder of an abstract summary model to obtain features of any target training sample; any target training sample and features of any target training sample are processed by a basic aspect module, an associated sentence module and a triple entity relationship module to obtain initial basic aspect features, initial associated sentence features and initial triple entity relationship features; the initial basic aspect features, initial associated sentence features and initial triple entity relationship features are respectively compared with target basic aspect features, target associated sentence features and target triple entity relationship features extracted by a structural principle of any target training sample to calculate cosine loss to minimize the cosine loss, wherein the structural principle of any target training sample is obtained by processing any target training sample with a large language model; the initial basic aspect features, initial associated sentence features and initial triple entity relationship features are respectively compared with the target basic aspect features, target associated sentence features and target triple entity relationship features extracted by a structural principle of any target training sample. The associated sentence features and the initial triple entity relationship features are input into their respective entry neural networks to obtain features corresponding to the initial basic aspect features, features corresponding to the initial associated sentence features, and features corresponding to the initial triple entity relationship features; the features corresponding to the initial basic aspect features, the features corresponding to the initial associated sentence features, and the features corresponding to the initial triple entity relationship features are fused with the features of different levels of the decoder of the abstract summary model, and a target summary of any target training sample is output, which solves the technical problems of low quality, controllability, interpretability and stability of the summary generated by extracting summaries from texts through LLMs in the prior art, and achieves the goal of converting the text summary generation task into a three-stage task, effectively utilizing the outstanding structured principle reasoning ability of LLM, and hierarchically distilling the structured principles to SLM, thereby improving the quality of its summary generation, as well as the technical effects of controllability, interpretability and stability.

[0103] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0104] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0106] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0107] In addition, each functional unit in each embodiment of the present invention may be integrated into a first processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0108] The above are only preferred embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.< / s> < / s> ​< / s> < / s> ​

Claims

1. A text summary generation method based on a large language model distillation framework, characterized in that: include: Obtain multiple target training samples; Process any target training sample through the encoder of the abstract summary model to obtain the features of any target training sample; Processing any target training sample and any target training sample's features through the basic aspect module, the associated sentence module and the triple entity relationship module to obtain initial basic aspect features, initial associated sentence features and initial triple entity relationship features; The initial basic aspect features, the initial associated sentence features and the initial triple entity relationship features are respectively compared with the target basic aspect features, the target associated sentence features and the target triple entity relationship features extracted by the structured principle of any target training sample to calculate the cosine loss so as to minimize the cosine loss, wherein the structured principle of any target training sample is obtained by processing any target training sample with a large language model; Inputting the initial basic aspect features, the initial associated sentence features, and the initial triple entity relationship features into respective entry neural networks, obtaining features corresponding to the initial basic aspect features, features corresponding to the initial associated sentence features, and features corresponding to the initial triple entity relationship features; The features corresponding to the initial basic aspect features, the features corresponding to the initial associated sentence features, and the features corresponding to the initial triple entity relationship features are fused with the features of different levels of the decoder of the abstract summary model to output a target summary of any target training sample.

2. The method according to claim 1, characterized in that: Before acquiring a plurality of target training samples, the method further includes: Acquire multiple initial training samples, wherein each initial training sample includes text and a standard summary; Through a large language model, each substructuring principle and a text portion of an initial training sample corresponding to each substructuring principle are processed to obtain multiple sub-summaries, wherein each substructuring principle corresponds to a sub-summary; Compress multiple sub-summaries through a large language model to obtain a total summary for each initial training sample; Based on the total summary and the standard summary of each initial training sample, the quality score of the structured principle of each initial training sample is obtained; The multiple initial training samples are screened based on the quality score of the structured principle of each initial training sample to obtain multiple target training samples.

3. The method according to claim 1, characterized in that The expression for calculating the cosine loss by comparing the initial basic aspect features, initial associated sentence features, and initial triple entity relationship features with the target basic aspect features, target associated sentence features, and target triple entity relationship features extracted by the structured principle of any target training sample is: L EA =yes out , > ​ L AS =yes out , <s>>< / s> ​ <s> L TER =yes <R out , <r> >< / r> < / s> <s> Among them, L EA is the cosine loss between the initial basic aspect feature and the target basic aspect feature, A out is the initial basic aspect feature, The basic characteristics of the target; L AS is the cosine loss between the initial associated sentence feature and the target associated sentence feature, S out is the initial associated sentence feature, <s>is the target associated sentence feature; L TER is the cosine loss between the initial triple entity relationship features and the target triple entity relationship features, R out is the initial triple entity relationship feature, <r> is the target triple entity relationship feature.< / r> < / s> <s> 4. The method according to claim 1, characterized in that The method further includes: outputting an expression of a composite loss function of a target summary of any target training sample as follows: Among them, L is the composite loss function, is the loss function of the abstract summary model, and γ1 and γ2 are two hyperparameters.

5. The method according to claim 2, characterized in that: Before processing the text portion of each initial training sample by each substructuring principle of the large language model to obtain a plurality of sub-summaries, the method further includes: Extract multiple topic words corresponding to each standard summary sentence in each initial training sample; Extracting multiple related sentences of multiple subject words corresponding to each standard summary sentence in each initial training sample; Extract triple entity relations from each associated sentence to obtain triple entity relations of each standard summary sentence; Based on the triple entity relationship of multiple subject words, multiple related sentences, and each standard summary sentence, a structuring principle of the standard summary of each initial training sample is obtained; wherein the triple entity relationship of a subject word, a related sentence, and a standard summary sentence is a substructuring principle.

6. The method according to claim 2, characterized in that The text part of each initial training sample is processed by each sub-structuring principle of the large language model to obtain the expression of multiple sub-summaries: in, is the sub-summary corresponding to the ith sentence, ~ is the probability distribution, S mul are multiple sub-summaries, D is each training sample, R is the triple entity relationship of each training sample, |R| represents the number of structured principles in the triple entity relationship of each training sample, represents the triple entity relationship corresponding to the i-th sentence, s mul,＜i Represents the sub-summary corresponding to the first i sentences.

7. The method according to claim 2, characterized in that The expression of the total summary of each initial training sample obtained by compressing multiple sub-summaries through a large language model is: in, is the total summary corresponding to the i-th sentence, S tol is the total summary for each initial training sample, s tol,＜i Represents the total summary corresponding to the first i sentences.

8. The method according to claim 2, characterized in that: Based on the total summary and standard summary of each initial training sample, the quality score of the structured principle of each initial training sample is obtained including: When each training sample is a long document, the quality score of the structured principle of each initial training sample is expressed as: Among them, n is used to represent the number of phrases in each sub-abstract of the total abstract of each initial training sample. The value range of n is 1 to 2. match (gram n ) indicates the number of phrases that match the words in the general abstract and each sub-abstract of the standard abstract. Count(gram n ) is the number of phrases n in each sub-abstract of the standard abstract, and Score1 represents the quality score of the structural principle when the initial training sample is a long document; When each training sample is a short document, the quality score of the structured principle of each initial training sample is expressed as: The value range of n is 1 to 3, and Score2 represents the quality score of the structured principle when the initial training sample is a long document.

9. A processor, characterized in that: The processor is used to run a program, wherein the program executes the method according to any one of claims 1 to 8 when running.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 8. < / s> < / s>

Images