Self-Supervised Abstract Generation Method, System, Device and Medium

Abstract

Embodiments of this application relate to the field of natural language processing technology, and disclose a self-supervised abstract generation method, system, device, and medium. Use a pre-trained language model to generate document-abstract pairs; screen the document-abstract pairs through a self-supervised filter module to obtain a data set; use an expert iteration module to perform multiple rounds of self-supervised learning on the data set to iterate the teacher model; distill the knowledge of the iterated teacher model into the student model; through a controllable abstract generation module, introduce control attributes in the training and inference phases, and use the student model to generate customized abstracts according to the control attributes. By constructing a self-supervised information-theoretic objective and combining it with a small language model for distillation and training, the dependence on large-scale pre-trained models is reduced. It can at least solve the technical problem of low-quality abstract generation.

Description

Technical Field

[0001] This application relates to the technical field of natural language processing, and particularly to a self-supervised abstract generation method, system, device and medium. Background Art

[0002] Automatic summarization technology is an important task in natural language processing (NLP), aiming to extract concise and informative content from the original text. Existing automatic summarization methods can be classified into the following categories:

[0003] Extractive summarization: This method selects some important sentences or paragraphs from the original text as the summary. Common techniques include those based on TF-IDF, TextRank algorithm, etc. The advantage of this method is that the generated summary is linguistically more consistent and easier to understand. However, its limitation is that it cannot perform effective content compression, and the generated summary may lack fluency and naturalness.

[0004] Generative summarization: Generative methods aim to understand the content of the original text and restate it in a more concise language. This method relies on deep learning models, especially sequence-to-sequence models (Seq2Seq) and pre-trained language models (such as BERT, GPT, etc.). Generative summary models have strong content compression capabilities and flexibility, and can generate more fluent and natural text. However, such methods usually require a large amount of manually annotated data for training, and the training process is complex and consumes a large amount of computing resources.

[0005] In practical applications, most generative summarization methods rely on large-scale pre-trained language models such as GPT-4 and ChatGPT, and rely on unsupervised or imitation learning for training. However, for what kind of summary is considered "good", the existing technology lacks a unified quantitative standard. Summary of the Invention

[0006] An object of this application is to provide a self-supervised abstract generation method, device, medium and product, at least to solve the technical problem of low-quality abstract generation.

[0007] To achieve the above object, some embodiments of this application provide the following aspects:

[0008] In a first aspect, some embodiments of the present application further provide a self-supervised abstract generation method, including: using a pre-trained language model to generate document-abstract pairs; screening the document-abstract pairs through a self-supervised filter module to obtain a data set; using an expert iteration module to perform multiple rounds of self-supervised learning on the data set to iterate the teacher model; distilling the knowledge of the iterated teacher model into a student model; and introducing control attributes in the training and inference phases through a controllable abstract generation module, and using the student model to generate customized abstracts according to the control attributes.

[0009] In a second aspect, some embodiments of the present application further provide a self-supervised abstract generation system, including: a pre-trained language model for generating document-abstract pairs; a self-supervised filter module for screening the document-abstract pairs to obtain a data set; an expert iteration module for performing multiple rounds of self-supervised learning on the data set to iterate the teacher model; distilling the knowledge of the iterated teacher model into a student model; and a controllable abstract generation module for introducing control attributes in the training and inference phases and using the student model to generate customized abstracts according to the control attributes.

[0010] In a third aspect, some embodiments of the present application further provide an electronic device, which includes: one or more processors; and a memory storing computer program instructions, and when the computer program instructions are executed, the processors execute the steps of the method as described above.

[0011] In a fourth aspect, some embodiments of the present application further provide a computer-readable medium, on which computer program instructions are stored, and the computer program instructions can be executed by a processor to implement the method as described above.

[0012] Compared with the related art, in the solution provided by the embodiments of the present application, by constructing a self-supervised information-theoretic objective and combining distillation and training with a small language model, not only the dependence on large-scale pre-trained models is reduced, but also the quality of the model is optimized through information maximization and expert iteration. In addition, by introducing control attributes and a self-supervised filtering mechanism, the controllability, quality, and cross-domain adaptability of the abstract are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplary illustrations do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements, unless otherwise stated, and the drawings in the figures do not constitute a scale limitation.

[0014] Figure 1 FIG. is a schematic flowchart of a self-supervised abstract generation method provided according to an embodiment of the present application;

[0015] Figure 2 It is a schematic structural diagram of a self-supervised abstract generation system provided according to an embodiment of the present application;

[0016] Figure 3 It is a schematic structural diagram of an electronic device provided according to an embodiment of the present application. Detailed implementation manners

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0018] First Embodiment

[0019] The first embodiment of the present application relates to a self-supervised abstract generation method. As Figure 1 shown, the method may include the following steps:

[0020] S101, generating document-abstract pairs using a pre-trained language model:

[0021] Provide a basic pre-trained language model. Although it does not directly have the dedicated abstract generation ability itself, it can generate preliminary documents and their corresponding abstracts based on its powerful language understanding and generation capabilities. This model is used as a teacher model to generate preliminary document-abstract pairs.

[0022] S102, screening the document-abstract pairs through a self-supervised filter module to obtain a dataset:

[0023] Use a self-supervised language model (such as the BERT model based on the masked language model MLM) to calculate the mutual information (PMI) between the document and the abstract. PMI measures the degree of association between the document and the abstract. Set the screening criteria from three aspects: significance, credibility, and conciseness:

[0024] Significance: Ensure that the abstract contains the key information in the document, and measure the significance by calculating the overlap degree between the keywords in the abstract and the keywords in the document. For example, set a threshold for the keyword overlap ratio. Only when the keyword overlap ratio between the abstract and the document exceeds this threshold is the abstract considered to have sufficient significance.

[0025] Credibility: Ensure that the abstract content is true and reliable, without containing fictional information. The credibility can be evaluated by checking whether the factual information in the abstract is consistent with the document. For example, for an abstract containing specific data and events, verify whether there is an accurate description corresponding to it in the document.

[0026] Simplicity: Control the length of the abstract and avoid including excessive redundant information. A ratio threshold of the abstract length to the document length can be set to filter out the abstracts that meet the length requirements.

[0027] Filter document-abstract pairs According to the calculated PMI values and the set filtering criteria, filter out the document-abstract pairs with PMI values higher than the preset threshold and meeting the requirements of significance, credibility, and simplicity to form a high-quality dataset.

[0028] S103, Use the expert iteration module to perform multiple rounds of self-supervised learning on the dataset to iterate the teacher model:

[0029] Use the expert iteration module to perform multiple rounds of self-supervised learning on the dataset to iteratively optimize the teacher model. The teacher model can choose a model with a Transformer architecture, such as the T5 model. In each round of self-supervised learning, input the document-abstract pairs in the dataset into the teacher model for forward propagation calculation. The teacher model will generate a predicted abstract based on the input document, and then compare the predicted abstract with the true abstract in the dataset to calculate the loss function between the two.

[0030] The commonly used loss function can choose the cross-entropy loss function. According to the calculated loss function, use the backpropagation algorithm to update the parameters of the teacher model. Through multiple rounds of self-supervised learning and parameter updates, gradually improve the abstract generation ability of the teacher model.

[0031] S104, Distill the knowledge of the iterated teacher model into the student model:

[0032] To reduce the computational cost and improve the inference efficiency of the model, use information distillation technology to distill the knowledge of the iterated teacher model into a small student model. The student model can choose a Transformer model with a smaller parameter scale, such as DistilBERT. Specifically, define the loss function between the outputs of the teacher model and the student model, which measures the difference between their outputs. The commonly used loss function can be the mean squared error loss function. By minimizing this loss function, adjust the parameters of the student model so that the output of the student model is as close as possible to the output of the teacher model, thereby transferring the knowledge of the teacher model to the student model.

[0033] S105. In the training and inference stages, introduce control attributes through the controllable abstract generation module, and use the student model to generate a customized abstract according to the control attributes:

[0034] Use the controllable abstract generation module to introduce control attributes in the training and inference stages of the model to generate a customized abstract that meets specific user needs. The control attributes can include the length of the abstract, the extractability of the abstract, the keywords that the abstract focuses on, etc.

[0035] In the training stage, input the control attributes together with the document-abstract pair into the student model. For example, for the control attribute of the abstract length, the expected abstract length can be encoded as a vector and input into the student model together with the document input. The model will learn the ability to generate appropriate abstracts under different control attributes.

[0036] In the inference stage, according to the control attributes input by the user, use the trained student model to generate a customized abstract that meets the requirements. For example, if the user hopes to generate an abstract with a length of 200 words and focuses on a specific keyword in the document, input these control attributes into the student model, and the student model will generate an abstract that meets the requirements based on this information.

[0037] It is not difficult to find that compared with the related technologies, in the solution provided by the embodiment of the present application, through the information distillation technology, the knowledge of the optimized teacher model is transferred to the small student model, reducing the dependence on the large pre-trained model. While maintaining the high quality of the abstract, the computational overhead of training and inference is greatly reduced, effectively solving the problem of high computational cost of traditional methods; with the help of the information maximization objective, the abstract generation is optimized from three aspects of significance, credibility and conciseness, and the teacher model is subjected to multiple rounds of self-supervised learning through the expert iteration mechanism, gradually improving the model generation ability, so that the generated abstract can accurately extract the key information of the document, faithfully reflect the original content and be concise; by introducing control attributes, users can flexibly specify personalized requirements such as the length of the abstract, extractability, keyword focus, etc. The model learns the generation rules under different control attributes in the training stage, and can dynamically adjust the generation strategy according to the user input during inference, significantly enhancing the controllability and flexibility of the abstract generation.

[0038] Second Embodiment

[0039] The second embodiment of the present application relates to a self-supervised abstract generation method. The second implementation is an improvement based on the first embodiment. The specific improvement lies in:

[0040] The screening of the document-abstract pairs by the self-supervised filter module includes: performing a masking process on the document-abstract pairs, calculating the mutual information between the document and the abstract, and calculating the saliency score, credibility score, and conciseness score of the abstract according to the mutual information; screening out the document-abstract pairs whose saliency score, credibility score, and conciseness score meet the preset thresholds, and forming the dataset.

[0041] The self-supervised filter module is used to screen and optimize the initially generated document-abstract pairs to ensure that they meet the requirements of saliency, credibility, and conciseness. Use the self-supervised language model (MLM) to perform a masking process on the document, calculate the mutual information (PMI) between the document and the abstract, and evaluate the saliency, credibility, and conciseness of the abstract.

[0042] Saliency: Calculate the correlation between the document and the abstract through the mutual information formula (PMI) to ensure that the abstract contains the key information in the document. Saliency score formula:

[0043] ;

[0044] x represents the document; y represents the abstract; [] is the indicator function (1 when the condition is true, otherwise 0); is the probability that the masked language model (MLM) predicts the masked part as the original content of x when a part of the document x is masked ( ) and the abstract y is known; is the probability that the MLM predicts the masked part as the original content of x when only the masked part of the document is known (without the information of the abstract y);

[0045] The formula determines the prominence of the abstract for the key information of the document by comparing the absolute value of the logarithm ratio of the prediction probabilities of the masked part of the document with and without the abstract y. If this value is greater than the threshold , then indicates that the abstract significantly contains the key information of the document; otherwise it is 0.

[0046] Faithfulness: Ensure that the abstract does not add any information that is not in the original text by the possibility of restoring the abstract content. Faithfulness score formula:

[0047] ;

[0048] is the probability that the MLM predicts the masked part as the original content of y when a part of the abstract y is masked and the document x is known; Only knowing the masked part of the abstract (without document x information), the MLM predicts the probability that the masked part is the original content of y.

[0049] This formula measures the degree of support of the document for the abstract content. If the absolute value of the logarithmic ratio of the predicted probabilities of the masked part of the abstract with and without document x is greater than the threshold , then indicates that the abstract content is reliable based on the document; otherwise it is 0.

[0050] Brevity: Controlled according to the length of the abstract to ensure that the abstract is concise and refined. Brevity scoring formula:

[0051] ;

[0052] is the length of the abstract (such as the number of words, characters), is the length of the document.

[0053] The formula judges the brevity by the ratio of the length of the abstract to the document. If ( is the preset length ratio threshold), then , indicating that the abstract is concise and has no excessive redundant information; otherwise it is 0.

[0054] Only retain the document-abstract pairs that meet the requirements of significance, credibility, and brevity. After screening, a high-quality dataset is obtained. Generally speaking, the core point is to calculate the PMI between the abstract y and the original text of x (that is, the probability of mutual information between the two), and whether the ratio of the scores between the two reaches a certain threshold. If it reaches, it means that the abstract is useful, and optimization has been carried out from three aspects.

[0055] Furthermore, the iterative teacher model includes: inputting the document-abstract pairs in the dataset into the teacher model for forward propagation calculation; calculating the first loss function according to the difference between the output result of the teacher model and the abstract; updating the parameters of the teacher model using the backpropagation algorithm according to the first loss function.

[0056] Optimize the teacher model through multiple rounds of iteration to make it generate higher-quality abstract data. Based on self-supervised learning, use the self-generated high-quality abstracts to train the teacher model to improve its generation ability in each round of iteration.

[0057] Optimization process: Assume that the current teacher model generates the initial document-abstract pair (x, y), and optimize the teacher model through the following steps:

[0058] Data generation: ;

[0059] Data screening: High-quality document-abstract pairs are screened out through a self-supervised filter to obtain a dataset;

[0060] Model optimization: Update the teacher model using the screened dataset:

[0061]

[0062] Represents the state of the teacher model after the (t + 1)-th iteration; Represents finding the teacher model that maximizes the subsequent expectation ; E(x, y) calculates the expectation for samples (x, y) in the high-quality dataset in; is the teacher model The logarithm of the probability of generating the abstract y given the document x. In each iteration, using the high-quality dataset , by maximizing the "expectation of the logarithm of the probability that the model generates the corresponding abstract y given the document x", the parameters of the teacher model are updated. After multiple iterations, the teacher model can better learn the ability to generate high-quality abstracts based on documents, improving the accuracy and quality of abstract generation.

[0063] Furthermore, distilling the knowledge of the iterated teacher model into the student model includes: defining a second loss function between the outputs of the teacher model and the student model, calculating the difference value between their outputs according to the second loss function; adjusting the parameters of the student model by minimizing the difference value of the loss function output.

[0064] Transfer the knowledge of the teacher model optimized through expert iteration to a smaller student model to achieve efficient inference. Through information distillation technology, the high-quality document-abstract pairs generated by the teacher model are used as training data to optimize the student model. Distillation process:

[0065] Generate data:

[0066] Train the student model: Optimize the student model through the following objective function:

[0067]

[0068] Represents the student model, which is a lightweight model designed to achieve the abstract generation function at a low computational cost; Represents in the parameter space of the student model to find the model parameter configuration that maximizes the subsequent expectation; E(x, Find the expectation of the sample set composed of), reflecting the comprehensive consideration of multiple groups of input and output samples; is the student model When given the document x, generate the summary produced by the teacher model The logarithm of the probability. By maximizing this value, the student model learns to imitate the summary generation behavior of the teacher model.

[0069] Using knowledge distillation technology, let the student model learn the summary generation ability of the teacher model. The generated by the teacher model is used as a supervision signal. The student model optimizes its own parameters to maximize the expected value of the logarithm of the probability of generating Thereby transferring the knowledge of the teacher model to itself, while maintaining a low computational cost, improving the summary generation performance. The student model can generate high-quality summaries similar to the teacher model with a relatively low computational overhead.

[0070] Furthermore, the control attributes include: summary length, information extractiveness, and keywords.

[0071] Generate summaries in a specific style according to user needs, such as controlling the length, extractiveness, or keyword focus of the summary. Control attribute tags: Add control attributes such as summary length, information extractiveness, keywords, etc. to the document-summary pairs during the training phase. Pass these control attributes as input conditions to the model to ensure that the generated summary meets the predetermined requirements.

[0072] Application examples of control attributes: Summary length control, control_attributes={length:medium}; Information extractiveness control, control_attributes={extractiveness:high}; Keyword focus control, control_attributes={focus_keyword:disease}. The generated summary y will meet the requirements of these control attributes, ensuring that the summary meets the user's needs.

[0073] Furthermore, using the student model to generate customized summaries according to the control attributes includes: obtaining the user input instruction, extracting the input instruction to obtain the control attributes; inputting the control attributes into the pre-trained language model, and passing them to the self-supervised filter module, the expert iteration module, and the controllable summary generation module to generate the customized summary.

[0074] The user will input an instruction containing specific requirements for generating an abstract, such as "Generate an abstract about the development of artificial intelligence that is about 200 words long and highlights the key technological innovation points". The system will perform natural language processing on the input instruction to extract the key control attributes. In the above example, the control attributes include "abstract length of about 200 words" and "highlight key technological innovation points".

[0075] The extracted control attributes are input into the pre-trained language model. The pre-trained language model will encode the control attributes and transform them into vector representations suitable for subsequent module processing. The self-supervised filter module will perform more accurate screening of the generated document-abstract pairs according to the control attributes. For example, if the control attribute requires the abstract length, this module will screen out the document-abstract pairs that meet the length requirement.

[0076] When training the teacher model, the expert iteration module will optimize the model in combination with the control attributes. For example, if the control attributes emphasize certain keywords, the model will pay more attention to the content related to these keywords. The controllable abstract generation module directly uses the control attributes to generate customized abstracts. It will adjust the strategy for generating the abstract according to the control attributes to ensure that the generated abstract meets the specific needs of the user.

[0077] Furthermore, using the student model to generate a customized abstract according to the control attributes includes:

[0078] In the training stage, according to the control attributes and the document-abstract pairs, guide the student model to learn to generate corresponding abstracts under the control attributes; in the inference stage, according to the control attributes, use the trained student model to generate the customized abstract.

[0079] In the training stage, the system will combine the control attributes with the document-abstract pairs. For example, for each document-abstract pair, input the corresponding control attributes (such as abstract length, keywords, etc.) at the same time. The student model will learn the ability to generate corresponding abstracts under different control attributes. Through a large number of training samples, the model will gradually master how to adjust the generation method of the abstract according to the control attributes to meet various needs.

[0080] In the inference stage, generate an abstract according to the control attributes. When receiving a new document and control attributes, use the trained student model to generate a customized abstract. The student model will generate a compliant abstract according to the input control attributes by applying the knowledge learned in the training stage.

[0081] It is not difficult to find that in the embodiments of the present application, by extracting control attributes from user inputs and utilizing these attributes in the training and inference stages, the system can generate customized summaries that meet the diverse needs of users. This method enhances the flexibility and practicality of summary generation, enabling the system to better adapt to applications in different scenarios.

[0082] The step divisions of the above various methods are only for clear description. When implemented, they can be combined into one step or some steps can be split into multiple steps. As long as the same logical relationship is included, they are all within the protection scope of this application; adding insignificant modifications or introducing insignificant designs to the algorithms or processes, but not changing the core designs of their algorithms and processes are all within the protection scope of this application.

[0083] Third Embodiment

[0084] The third embodiment of the present application relates to a self-supervised summary generation system / device. As Figure 2 shown, the system includes:

[0085] A pre-trained language model for generating document-summary pairs;

[0086] A self-supervised filter module for screening the document-summary pairs to obtain a data set;

[0087] An expert iteration module for performing multiple rounds of self-supervised learning on the data set to iterate the teacher model; distilling the knowledge of the iterated teacher model into the student model;

[0088] A controllable summary generation module for introducing control attributes in the training and inference stages and using the student model to generate customized summaries according to the control attributes.

[0089] It is not difficult to find that this embodiment is a system embodiment corresponding to the first embodiment, and this embodiment can be implemented in cooperation with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment. To avoid repetition, they will not be elaborated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.

[0090] It is worth mentioning that each module involved in this embodiment is a logical module. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or can be implemented as a combination of multiple physical units. In addition, to highlight the innovative part of this application, units that are not closely related to solving the technical problems proposed in this application are not introduced in this embodiment, but this does not mean that there are no other units in this embodiment.

[0091] In addition, some embodiments of the present application further provide an electronic device. The electronic device may be various forms of digital computers, such as, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and so on. The electronic device may also be various forms of mobile devices, such as, personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices.

[0092] The electronic device includes: one or more processors; and a memory storing computer program instructions, which when executed cause the processor to perform the steps of the method provided in any one or more of the above embodiments. Figure 3 An exemplary structural diagram of the electronic device is disclosed. As Figure 3 shown, the electronic device includes: one or more processors 1101, a memory 1102, and interfaces for connecting the components, including a high-speed interface and a low-speed interface. Each component is interconnected using different buses and can be installed on a common motherboard or otherwise installed as required. The processor can process instructions executed within the electronic device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system). Among them, the components, their connections and relationships, and their functions shown herein are only examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0093] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103, and the output device 1104 can be connected through a bus or other means, Figure 3 taking the connection through the bus as an example.

[0094] The input device 1103 can receive input digital or character information and generate key signal inputs related to the user settings and function controls of the electronic device, such as input devices like touchscreens, keypads, mice, trackpads, touchpads, pointing sticks, one or more mouse buttons, trackballs, joysticks, etc. The output device 1104 can include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors), etc. The display device can include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device can be a touchscreen.

[0095] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide inputs to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback); and inputs from the user can be received in any form (including acoustic input, voice input, or haptic input).

[0096] In the embodiments of this application, computer programs / instructions are stored on a computer-readable medium. When the computer programs / instructions are executed by a processor, the steps of the methods provided in any one or more of the above embodiments are implemented. The computer-readable medium can be included in the electronic device described in the above embodiments; or it can exist separately without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.

[0097] The memory 1102 can be used as a non-transitory computer-readable storage medium and can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the methods provided in any one or more of the above embodiments of this application.

[0098] The memory 1102 may include a program storage area and a data storage area. The program storage area may store an operating system and application programs required for at least one function. The data storage area may store data created according to the use of the electronic device and the like. In addition, the memory 1102 may include a high-speed random access memory and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 1102 may optionally include a memory remotely disposed relative to the processor 1101, and these remote memories may be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0099] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the computer-readable medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0100] The computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information may be computer-readable instructions, data structures, program modules, or other data. Examples of the computer's storage medium include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other memory technologies, read-only optical discs (CD-ROMs), digital versatile discs (DVDs), or other optical storage, magnetic cassette tapes, magnetic tape disk storage, or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0101] Computer program code for performing the operations of the present application can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, by using an Internet service provider to connect through the Internet).

[0102] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. For example, a dedicated integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device can be used. In some embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive, or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to execute each step or function.

[0103] The computer program product provided by the embodiments of the present application includes one or more computer programs / instructions. When the computer programs / instructions are executed by a processor, they wholly or partly generate the processes or functions described in the embodiments of the present application. The computer can be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

[0104] The flowcharts or block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of devices, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0105] The scope of the present application is defined by the appended claims rather than the above description. Therefore, all changes that fall within the meaning and scope of the equivalent elements of the claims are intended to be encompassed within the present application. Any reference numerals in the claims should not be construed as limiting the claims involved. In addition, it is obvious that the term "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices stated in the apparatus claims may also be implemented by one unit or device through software or hardware. The terms "first", "second", etc. are only used for descriptive distinction and do not represent any specific order, nor can they be construed as indicating or implying relative importance.

[0106] As described above, this is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily mention changes or substitutions, which should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.

Claims

1. A self-supervised summary generation method, characterized in that The method comprises: Generate document-summary pairs using a pre-trained language model; Filtering the document-summary pairs through a self-supervised filter module to obtain a dataset; Use the expert iteration module to perform multiple rounds of self-supervised learning on the dataset to iterate the teacher model; Distill the knowledge of the iterative teacher model into the student model; Through a controllable summary generation module, control attributes are introduced during the training and inference phases of the student model, and customized summaries are generated using the student model according to the control attributes, wherein the control attributes include summary length, information extractivity, and keywords; The screening of the document-summary pairs by the self-supervisory filter module includes: Masking the document-abstract pair, calculating the mutual information between the document and the abstract, and calculating the significance score, credibility score, and conciseness score of the abstract based on the mutual information; The document-summary pairs whose significance scores, credibility scores, and conciseness scores meet preset thresholds are screened out to form the dataset.

2. The method according to claim 1, characterized in that The iterative teacher model includes: Input the document-summary pairs in the data set into the teacher model for forward propagation calculation; Calculating a first loss function based on the difference between the output of the teacher model and the summary; According to the first loss function, the parameters of the teacher model are updated using a backpropagation algorithm.

3. The method according to claim 2, characterized in that The step of distilling the iterative knowledge of the teacher model into the student model includes: Defining a second loss function between the outputs of the teacher model and the student model, and calculating the difference between the outputs of the teacher model and the student model according to the second loss function; The parameters of the student model are adjusted by minimizing the difference value output by the second loss function.

4. The method according to claim 1, wherein Generating a customized summary using the student model according to the control attribute includes: Obtaining a user input instruction, extracting the input instruction, and obtaining the control attribute; The control attributes are input into a pre-trained language model and passed to the self-supervisory filter module, the expert iteration module and the controllable summary generation module to generate the customized summary.

5. The method according to claim 4, characterized in that Generating a customized summary using the student model according to the control attributes includes: In a training phase, based on the control attribute and the document-summary pair, the student model is guided to learn to generate a corresponding summary under the control attribute; In the inference phase, the customized summary is generated using the trained student model according to the control attributes.

6. A self-supervised summary generation system, characterized in that The system comprises: Pre-trained language model for generating document-summary pairs; The self-supervised filter module is used to filter the document-summary pairs to obtain a data set, specifically including: Masking the document-abstract pair, calculating the mutual information between the document and the abstract, and calculating the significance score, credibility score, and conciseness score of the abstract based on the mutual information; Screening out document-abstract pairs whose significance scores, credibility scores, and conciseness scores meet preset thresholds to form the dataset; An expert iteration module is used to perform multiple rounds of self-supervised learning on the dataset, iterate the teacher model, and distill the knowledge of the iterated teacher model into the student model; A controllable summary generation module is used to introduce control attributes during the training and inference phases of the student model, and to generate a customized summary using the student model according to the control attributes, wherein the control attributes include summary length, information extractivity, and keywords.

7. An electronic device, characterized in that: The electronic device comprises: one or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images