A conference summary generation method and system based on pre-training and prompting

Abstract

The present invention discloses a method and system for generating conference summaries based on pre-training and prompts, the method comprising: obtaining conference text data and performing mapping and segmentation processing to obtain a fragmented conference text vector sequence; pre-training a Transformer-XL model based on an unlabeled text data set to obtain a trained Transformer-XL model; based on a task prompt method, inputting the fragmented conference text vector sequence into the trained Transformer-XL model to obtain a conference summary. By using the present invention, the contextual information of the conference can be predicted to generate a conference summary that is more in line with the conference information. As a method and system for generating conference summaries based on pre-training and prompts, the present invention can be widely used in the field of natural language processing technology.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and system for generating conference summaries based on pre-training and prompting. Background Art

[0002] Meeting summary refers to a method of extracting key points from meeting minutes to present the content of the meeting in a concise form. The meeting summary can be further organized into meeting minutes to convey the meeting situation and agreed matters. The current mainstream automatic meeting summarization technology is mainly divided into extractive summarization methods represented by TextRank and generative methods represented by the improved Transformer algorithm. These methods all take text-based meeting minutes as input and output text summaries of the meeting minutes. However, the extractive summarization method represented by TextRank directly sources the summary from the truncation of the meeting minutes text, so that the generated summary is prone to missing important information and does not include further understanding and reasoning. Especially for meeting minutes that are mainly in the form of multi-person dialogues, the extractive summarization method cannot obtain the information contained in the communication between the participants, and can only extract some obvious sentences. The resulting summary is not flexible enough, and the lack of understanding and reasoning may cause some of the views in the resulting summary to be inconsistent with the actual expression of the meeting. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide a conference summary generation method and system based on pre-training and prompting, which can predict the context information of the conference and generate a conference summary that is more consistent with the conference information.

[0004] The first technical solution adopted by the present invention is: a method for generating conference summaries based on pre-training and prompting, comprising the following steps:

[0005] Acquire conference text data and perform mapping and segmentation processing to obtain a fragmented conference text vector sequence;

[0006] Pre-train the Transformer-XL model based on the unlabeled text dataset to obtain the trained Transformer-XL model;

[0007] Based on the task prompt method, the fragmented meeting text vector sequence is input into the trained Transformer-XL model to obtain the meeting summary.

[0008] Furthermore, the step of acquiring the conference text data and performing mapping and segmentation processing to obtain a fragmented conference text vector sequence specifically includes:

[0009] Collect audio data of the meeting through audio equipment;

[0010] The audio data of the conference is converted and processed through the API interface to obtain the conference text data;

[0011] Mapping the conference text data to obtain a conference text vector sequence;

[0012] The conference text vector sequence is segmented to obtain a fragmented conference text vector sequence.

[0013] Furthermore, the step of mapping the conference text data to obtain a conference text vector sequence specifically includes:

[0014] The conference text data is split according to the characters of the conference text data to obtain a conference text character sequence;

[0015] Constructing a dictionary, the dictionary including numeric indices mapping conference text character sequences;

[0016] Assign the numerical index in the dictionary to the conference text character sequence and express it to obtain the conference text vector sequence.

[0017] Furthermore, the step of pre-training the Transformer-XL model based on the unlabeled text dataset to obtain the trained Transformer-XL model specifically includes:

[0018] Obtain an unlabeled text dataset for data preprocessing to obtain a training set;

[0019] Inputting the training set into a Transformer-XL model, wherein the Transformer-XL model includes an input layer, a hidden layer, and an output layer;

[0020] Based on the input layer of the Transformer-XL model, the training set is processed by word embedding and position embedding to obtain the hidden state vector of the training set;

[0021] Based on the hidden layer of the Transformer-XL model, the hidden state vector of the training set is processed by data hiding operation to obtain a hidden vector with context information;

[0022] Based on the output layer of the Transformer-XL model, the latent vector with context information is projected to obtain the output result;

[0023] Based on the output results, the Transformer-XL model is updated by the gradient descent method to obtain the trained Transformer-XL model.

[0024] Furthermore, the expression of the hidden layer of the Transformer-XL pre-trained model is as follows:

[0025]

[0026] In the above formula, represents the hidden state vector output by the τth fragment token sequence after passing through the jth layer of the Transformer-XL model, W o represents the output projection matrix, and represents the projection matrix, represents the inverse matrix of a k-dimensional all-one vector, and μ represents an arbitrary constant.

[0027] Furthermore, the expression of the output layer of the Transformer-XL pre-trained model is as follows:

[0028]

[0029] In the above formula, W u represents the projection matrix, Indicates yes The hidden state vector output after passing through m identical hidden layers, Y represents the subsequent text token sequence output, and X represents the text token sequence representing the text input.

[0030] Furthermore, the task prompt-based method inputs the fragmented conference text vector sequence into the trained Transformer-XL model to obtain the conference summary, which specifically includes:

[0031] Obtain the meeting summary dataset and input it into the trained Transformer-XL model for fine-tuning to obtain task prompts;

[0032] Based on the task prompts, the fragmented conference text vector sequences are sequentially input into the trained Transformer-XL model to obtain the vector sequence of the conference summary text.

[0033] The vector sequence of the conference summary text is converted to obtain the conference summary.

[0034] Furthermore, the step of obtaining the meeting summary dataset and inputting it into the trained Transformer-XL model to obtain the task prompt specifically includes:

[0035] Obtain the conference summary dataset and perform data preprocessing to obtain the conference summary vector;

[0036] Set a constant matrix of the context range of the meeting summary task and perform mapping processing through a mapping matrix to obtain an initial memory vector sequence;

[0037] Input the initial memory vector sequence and the meeting summary vector into the trained Transformer-XL model to obtain the predicted value;

[0038] Based on the predicted value, the parameters of the mapping matrix are updated by the gradient descent method, and finally the task prompt vector is obtained.

[0039] Furthermore, based on the task prompt, the step of sequentially inputting the fragmented conference text vector sequence into the trained Transformer-XL model to obtain the vector sequence of the conference summary text specifically includes:

[0040] After the single hidden layer fully connected neural network model is trained with data, an initial memory vector is generated;

[0041] Concatenate the initial memory vector with the fragmented conference text vector sequence to obtain a concatenated vector;

[0042] The concatenated vectors update the initial memory vectors respectively and are input into the trained Transformer-XL model;

[0043] The trained Transformer-XL model is guided by task prompts to output the hidden state vector of the hidden layer;

[0044] The hidden state vector of the hidden layer is concatenated with the updated memory vector and input into the trained Transformer-XL model for traversal training;

[0045] Determine whether the number of traversals reaches the number of hidden layers of the trained Transformer-XL model;

[0046] If it is determined that the number of layers requirement is not met, the above steps of splicing, updating and inputting the model are repeated until the number of layers requirement is met and all fragmented conference text vector sequences are traversed to output the vector sequence of the conference summary text.

[0047] The second technical solution adopted by the present invention is: a conference summary generation system based on pre-training and prompting, comprising:

[0048] An acquisition module acquires conference text data and performs mapping and segmentation processing to obtain a fragmented conference text vector sequence;

[0049] The training module is used to pre-train the Transformer-XL model based on the unlabeled text dataset to obtain the trained Transformer-XL model;

[0050] The output module, based on the task prompt method, inputs the fragmented conference text vector sequence into the trained Transformer-XL model to obtain the conference summary.

[0051] The beneficial effects of the method and system of the present invention are as follows: the present invention constructs a Transformer-XL model and uses a large amount of conference texts for pre-training, so that the pre-trained Transformer-XL model has the ability to process information in segments, and further obtains the summary of the conference text to fine-tune the trained Transformer-XL model, so that the pre-trained model is given prompts about the summary task, relevant knowledge can be derived from the pre-trained model, and the computing power cost of model training is saved. A generative summary method is adopted, which can understand and infer the dialogue in the conference record text, so that the generated summary is more in line with the conference information. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of the steps of a method for generating conference summaries based on pre-training and prompting of the present invention;

[0053] Figure 2 It is a structural block diagram of a conference summary generation system based on pre-training and prompting of the present invention;

[0054] Figure 3 It is a flowchart of the steps of generating a summary after obtaining a task prompt by the Transformer-XL pre-trained model of the present invention. DETAILED DESCRIPTION

[0055] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0056] Reference Figure 1 The present invention provides a method for generating conference summaries based on pre-training and prompting, the method comprising the following steps:

[0057] S1. Acquire conference text data and perform mapping and segmentation processing to obtain a fragmented conference text vector sequence;

[0058] Specifically, a meeting is recorded in real time until the end of the meeting, and a recording file of the entire meeting content is obtained. The speech-to-text API service interface provided by the network platform is directly called (here, the API service provided on the Internet is directly called, such as iFlytek speech transcription, to convert the recording of the entire meeting content into plain text, and each character of the plain text string is separated to form a character sequence, and a dictionary is constructed to map the characters of the string type to a digital index starting from 0. According to the dictionary, each unique character is assigned a digital index, and the character sequence of the text string is converted from a string representation to a digital index representation to obtain the final token sequence. The token sequence X = (x1, x2, ..., x n ) is split according to the following formula, and its expression is as follows:

[0059]

[0060] In the above formula, k represents the size of the segmentation, x i ∈X, So we can get the fragment sequence X′=(x1,x2,…,x n′ ).

[0061] S2. Pre-train the Transformer-XL model based on the unlabeled text dataset to obtain a trained Transformer-XL model;

[0062] Specifically, a large-scale unlabeled data set is established (usually text data collected from the Internet such as Baidu Knows, Zhihu Q&A, Baidu Encyclopedia, etc.), and data cleaning operations such as deduplication and removal of missing values ​​are performed on the data. The text is preprocessed and converted into a token sequence. The Transformer-XL autoregressive model, i.e., the Transformer-XL pre-training model, is established. Its task can be formalized as follows: given a token sequence X = (x1, x2, …, x n ), output the token sequence Y of the subsequent text = (y1, y2, ..., y n ), then we get the token sequence Y = (y1, y2, …, y n ) has the highest probability, and finally the token sequence Y=(y1,y2,…,y n );

[0063] S21. Train the input layer of the Transformer-XL pre-trained model.

[0064] Specifically, the token sequence X of the entire input text is (x1, x2, ..., x n), split into fixed-size, shorter segments and input them one by one into the Transformer-XL model. Let the length of the segment be k. For the τth segment token sequence x τ =(x τ,1 ,x τ,2 ,…,x τ,k ), in the input layer of the Transformer-XL model, the following operations are performed:

[0065]

[0066] In the above formula, Represents the representative token sequence x output by the input layer τ =(x τ,1 ,x τ,2 ,…,x τ,k ), the hidden state vector W e Represents the word embedding matrix of token, W p Represents the position embedding matrix of the token, W e , W p Represents the trainable parameters of the input layer.

[0067] S22, train the hidden layer of the Transformer-XL pre-trained model;

[0068] Specifically, the Transformer-XL model contains a set of m identical hidden layers. For the j∈{2,…,m+1}th layer of the Transformer-XL model, we have:

[0069]

[0070] In the above formula, represents the hidden state vector output by the τth fragment token sequence after passing through the jth layer of the Transformer-XL model, W o represents the output projection matrix, and represents the projection matrix, represents the inverse matrix of a k-dimensional all-one vector, and μ represents an arbitrary constant.

[0071] S23, train the output layer of the Transformer-XL pre-trained model;

[0072] Specifically, for the Transformer-XL model output layer, there is the following training expression:

[0073]

[0074] In the above formula, W u represents the projection matrix, Indicates yes The hidden state vector output after passing through m identical hidden layers, Y represents the subsequent text token sequence output, and X represents the text token sequence representing the text input;

[0075] Establish the objective function max φ P(Y|X; φ), where φ represents all trainable parameters of the Transformer-XL model. The model parameters φ are updated using the gradient descent method, and then:

[0076]

[0077] In the above formula, φ represents the updateable parameters of the Transformer-XL model;

[0078] After multiple rounds of iterations, the trained Transformer-XL model parameter φ can be obtained, that is, the pre-trained Transformer-XL model can be obtained.

[0079] S3. Based on the task prompt method, the fragmented meeting text vector sequence is input into the trained Transformer-XL model to obtain the meeting summary.

[0080] S31, get prompt, i.e., meeting task prompt;

[0081] Specifically, refer to Figure 3 , perform data cleaning operations such as deduplication and removal of missing values ​​on the conference summary dataset, and perform text preprocessing to convert it into a token sequence. For each piece of data, the token sequence x represents the text of the meeting minutes, which is the input of the model, and the token sequence y represents the content of the conference summary summarized by relevant professionals based on the meeting minutes. Let's assume represents a scope of context describing the summary task, then there exists:

[0082]

[0083] In the above formula, W θ represents the mapping matrix, B θ represents the mapping matrix, l represents the lth layer of the Transformer-XL pre-trained model, represents the initial memory vector;

[0084] Where W θ , B θ is the training parameter of the single hidden layer fully connected neural network model, the initial memory vector sequence is the initial input to the Transformer-XL pre-trained model;

[0085] The initial memory vector sequence Input Transformer-XL pre-trained model with sequence x The predicted value y′ can be obtained, so the objective function can be established as follows:

[0086]

[0087] In the above formula, z = [x; y] means concatenating x and y, i represents the time step, and Y idz Indicates the context scope of y, h i Represents the context of all history;

[0088] The autoregressive Transformer-XL pre-trained model converts h i Calculated as i and its left past context h ＜i The function is:

[0089]

[0090] In the above formula, P idx Represented by the parameter matrix, P θ The context range represented by h i (for all i) is the learnable parameter matrix P θ function, when i∈P idx When i Directly from P θ ,when When i Still dependent on P θ , because P θ The initial memory vector provided is always in the context of the left (i.e. historical) context, and therefore affects the representation of any subsequent context to its right;

[0091] The model parameter θ is updated using the gradient descent method. The updated expression is as follows:

[0092]

[0093] After multiple rounds of iterations, the trained parameter matrix P can be obtained θ , we can further obtain the initial memory vector sequence representing the context of the prompt summary task

[0094] S32. After obtaining the prompt, the Transformer-XL pre-trained model sequentially inputs a sequence of segments representing the segmented text of the meeting content;

[0095] Specifically, for the token sequence x1 of the first text segment, after passing through the input layer of the Transformer-XL neural network, the input layer hidden state vector can be obtained: The input layer hidden state vector of the Transformer-XL neural network in the memory vector sequence After concatenation, it is used as the input of the next layer of the Transformer-XL pre-trained model, as shown below:

[0096]

[0097] In the above formula, l∈{1,2,…,L} represents the lth layer of the Transformer-XL pre-trained model. When τ-1=0, it means that the hidden state vector comes from the initial memory vector sequence. layer(·) represents the specific operation of the layer of the Transformer-XL pre-trained model. At the same time, the vector representing the output hidden state of the lth layer of the Transformer-XL pre-trained model in the memory vector sequence is directly replaced by

[0098] The τth fragment token sequence x of the input τ After L layers of the Transformer-XL pre-trained model, the hidden state vector output by the last hidden layer of the Transformer-XL pre-trained model is obtained. And the updated memory vector sequence

[0099] In the fragment sequence X′=(x1,x2,…,x n′ After each segment of ) is operated according to the above steps, the last hidden state vector of the last segment is obtained And the updated memory vector sequence

[0100] S33, after the Transformer-XL pre-trained model inputs the token sequence representing the segmented text of the conference content, it continues to generate tokens of the conference summary in an autoregressive manner;

[0101] Specifically, the hidden state vector Input the output layer of the Transformer-XL neural network to obtain the distribution of the first summary text token, and then obtain the first summary text token;

[0102] The above steps will transform the hidden state vector Input the output layer of the Transformer-XL neural network to obtain a summary text token and memory vector sequence Input to the Transformer-XL pre-trained model, and you can get the distribution of the next summary text token. And the current latest memory vector sequence

[0103] Repeat the above steps until the model generates a token representing the end symbol. At this point, the token sequence formed from the first summary text token to the token representing the end symbol is the token sequence of the summary of the entire meeting content. The token sequence of the meeting summary text is converted back to text form, and finally the summary text of the entire meeting content can be obtained.

[0104] Reference Figure 2 , a conference summary generation system based on pre-training and prompting, comprising:

[0105] An acquisition module acquires conference text data and performs mapping and segmentation processing to obtain a fragmented conference text vector sequence;

[0106] The training module is used to pre-train the Transformer-XL model based on the unlabeled text dataset to obtain the trained Transformer-XL model;

[0107] The output module, based on the task prompt method, inputs the fragmented conference text vector sequence into the trained Transformer-XL model to obtain the conference summary.

[0108] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0109] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A conference summary generation method based on pre-training and prompting, characterized in that: The following steps are involved: Acquire conference text data and perform mapping and segmentation processing to obtain a fragmented conference text vector sequence; Pre-train the Transformer-XL model based on the unlabeled text dataset to obtain the trained Transformer-XL model; Based on the task prompt method, the fragmented conference text vector sequence is input into the trained Transformer-XL model to obtain the conference summary; The task prompt-based method inputs the fragmented conference text vector sequence into the trained Transformer-XL model to obtain the conference summary, which specifically includes: Obtain the meeting summary dataset and input it into the trained Transformer-XL model for fine-tuning to obtain task prompts; Based on the task prompts, the fragmented conference text vector sequences are sequentially input into the trained Transformer-XL model to obtain the vector sequence of the conference summary text. Convert the vector sequence of the conference summary text to obtain the conference summary; The step of obtaining the meeting summary dataset and inputting it into the trained Transformer-XL model to obtain the task prompt specifically includes: Obtain the conference summary dataset and perform data preprocessing to obtain the conference summary vector; Set a constant matrix of the context range of the meeting summary task and perform mapping processing through a mapping matrix to obtain an initial memory vector sequence; Input the initial memory vector sequence and the meeting summary vector into the trained Transformer-XL model to obtain the predicted value; Based on the predicted value, the mapping matrix is ​​updated by gradient descent method, and finally the task prompt vector is obtained; The step of inputting the fragmented conference text vector sequence into the trained Transformer-XL model in sequence based on the task prompt to obtain the vector sequence of the conference summary text specifically includes: After fine-tuning the method based on the task prompt, the initial memory vector is obtained; Concatenate the initial memory vector with the fragmented conference text vector sequence to obtain a concatenated vector; The concatenated vectors update the initial memory vectors respectively and are input into the trained Transformer-XL model; The trained Transformer-XL model is guided by task prompts to output the hidden state vector of the hidden layer; The hidden state vector of the hidden layer is concatenated with the updated memory vector and input into the trained Transformer-XL model for traversal; Determine whether the number of traversals reaches the number of hidden layers of the trained Transformer-XL model; If it is determined that the number of layers requirement is not met, the above steps of splicing, updating and inputting the model are repeated until the number of layers requirement is met and all fragmented conference text vector sequences are traversed to output the vector sequence of the conference summary text.

2. According to claim 1, a method for generating conference summaries based on pre-training and prompting, characterized in that: The step of acquiring conference text data and performing mapping and segmentation processing to obtain a fragmented conference text vector sequence specifically includes: Collect audio data of the meeting through audio equipment; The audio data of the conference is converted and processed through the API interface to obtain the conference text data; Mapping the conference text data to obtain a conference text vector sequence; The conference text vector sequence is segmented to obtain a fragmented conference text vector sequence.

3. According to claim 2, a method for generating conference summaries based on pre-training and prompting, characterized in that: The step of mapping the conference text data to obtain a conference text vector sequence specifically includes: The conference text data is split according to the characters of the conference text data to obtain a conference text character sequence; Constructing a dictionary, the dictionary including numeric indices mapping conference text character sequences; Assign the numerical index in the dictionary to the conference text character sequence and express it to obtain the conference text vector sequence.

4. According to claim 3, a method for generating conference summaries based on pre-training and prompting is characterized in that: The step of pre-training the Transformer-XL model based on the unlabeled text dataset to obtain the trained Transformer-XL model specifically includes: Obtain an unlabeled text dataset for data preprocessing to obtain a training set; Inputting the training set into a Transformer-XL model, wherein the Transformer-XL model includes an input layer, a hidden layer, and an output layer; Based on the input layer of the Transformer-XL model, the training set is processed by word embedding and position embedding to obtain the hidden state vector of the training set; Based on the hidden layer of the Transformer-XL model, the hidden state vector of the training set is processed by data hiding operation to obtain a hidden vector with context information; Based on the output layer of the Transformer-XL model, the latent vector with context information is projected to obtain the output result; Based on the output results, the Transformer-XL model is updated by the gradient descent method to obtain the trained Transformer-XL model.

5. A method for generating conference summaries based on pre-training and prompting according to claim 4, characterized in that: The expression of the hidden layer of the Transformer-XL pre-trained model is as follows: In the above formula, represents the hidden state vector output by the τth fragment token sequence after passing through the jth layer of the Transformer-XL model, W o represents the output projection matrix, and represents the projection matrix, represents the inverse matrix of a k-dimensional all-one vector, and μ represents an arbitrary constant.

6. According to claim 4, a method for generating conference summaries based on pre-training and prompting is characterized in that: The expression of the output layer of the Transformer-XL pre-trained model is as follows: In the above formula, W u represents the projection matrix, Indicates yes The hidden state vector output after passing through m identical hidden layers, Y represents the subsequent text token sequence output, and X represents the text token sequence representing the text input. Represents the inverse matrix of a k-dimensional all-one vector.

7. A conference summary generation system based on pre-training and prompting, characterized in that: Used to execute the conference summary generation method based on pre-training and prompting as claimed in claim 1, comprising the following modules: An acquisition module acquires conference text data and performs mapping and segmentation processing to obtain a fragmented conference text vector sequence; The training module is used to pre-train the Transformer-XL model based on the unlabeled text dataset to obtain the trained Transformer-XL model; The output module, based on the task prompt method, inputs the fragmented conference text vector sequence into the trained Transformer-XL model to obtain the conference summary.

Images