Text abstract generation model training method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a training method and device of a text abstract generation model, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a sub-word index sequence of a text sample; processing the sub-word index sequence to obtain a coding tensor sequence; according to the coding tensor sequence, carrying out internal loop training and external loop training on a preset TTT-Learar-Transformer model, and obtaining a target loss function; according to the target loss function, weight updating is carried out on the TTT-Learar-Transform model in a dual form until the TTT-Learar-Transform model converges, a text abstract generation model is obtained, and the text abstract generation model is used for generating an abstract of a text. According to the embodiment of the invention, the performance of the text abstract generation model can be improved, and the text abstract generation quality and efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, device, equipment and storage medium for a text summary generation model. Background Art

[0002] Text summarization refers to the key information extracted from text. It is concise and highly readable, and can help users quickly understand and obtain the required information. Generating text summaries through natural language processing (NLP) technology is an important research hotspot.

[0003] In related technologies, the text summary generation task is usually achieved using Seq2Seq technology, Gap Sentence Generation (GSG) technology and traditional Transformer model.

[0004] However, Seq2Seq and GSG technologies perform poorly on long texts and cannot effectively capture the relationship between distant words, resulting in the lack of coherence and logic in the generated text summaries, affecting the quality of text summary generation. Although the traditional Transformer model can effectively capture information in long texts through the self-attention mechanism, context loss may occur, resulting in the failure to capture deep key information. In addition, the traditional Transformer model has high computational complexity, which affects the quality and efficiency of text summary generation.

[0005] Therefore, how to improve the quality and efficiency of text summary generation has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The main purpose of the embodiments of the present application is to propose a training method, device, equipment and storage medium for a text summary generation model, aiming to improve the performance of the text summary generation model and improve the quality and efficiency of text summary generation.

[0007] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a training method for a text summary generation model, the method comprising:

[0008] Get the subword index sequence of the text sample;

[0009] Processing the subword index sequence to obtain an encoding tensor sequence;

[0010] Performing inner loop training and outer loop training on a preset TTT-Linear-Transformer model according to the encoded tensor sequence to obtain a target loss function of the TTT-Linear-Transformer model;

[0011] According to the target loss function, the weights of the TTT-Linear-Transformer model are updated in a dual form until the TTT-Linear-Transformer model converges, thereby obtaining a text summary generation model, which is used to generate a summary of the text.

[0012] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a training device for a text summary generation model, the device comprising:

[0013] A sample acquisition module is used to obtain a subword index sequence of a text sample;

[0014] An encoding processing module, used for processing the subword index sequence to obtain an encoding tensor sequence;

[0015] A model training module, used to perform inner loop training and outer loop training on a preset TTT-Linear-Transformer model according to the encoded tensor sequence, and obtain a target loss function of the TTT-Linear-Transformer model;

[0016] A weight updating module is used to update the weights of the TTT-Linear-Transformer model in a dual form according to the target loss function until the TTT-Linear-Transformer model converges to obtain a text summary generation model, and the text summary generation model is used to generate a summary of the text.

[0017] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0018] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0019] The present application proposes a training method, device, equipment and storage medium for a text summary generation model. The training method for the text summary generation model obtains a subword index sequence of a text sample; processes the subword index sequence to obtain a coding tensor sequence; performs inner loop training and outer loop training on a preset TTT-Linear-Transformer model according to the coding tensor sequence to obtain a target loss function of the TTT-Linear-Transformer model; according to the target loss function, updates the weight of the TTT-Linear-Transformer model in a dual form until the TTT-Linear-Transformer model converges to obtain a text summary generation model, which is used to generate a summary of the text. In this way, on the one hand, since the inner loop training provides local optimization in the summary generation process, the outer loop training ensures the global optimization of the TTT-Linear-Transformer model. Through the coordinated training of the inner loop and the outer loop, the TTT-Linear-Transformer model can handle short-term and long-term dependencies at the same time, so that it can efficiently and effectively learn to generate accurate and fluent summaries from the context, and realize the optimization training of the TTT-Linear-Transformer model for summary generation learning at the micro and macro levels; on the other hand, by introducing the dual form in the TTT-Linear-Transformer model, the TTT-Linear-Transformer model can more effectively utilize the information of the encoding tensor sequence, thereby reducing unnecessary calculations and improving training efficiency. As a result, the training effect is improved, thereby greatly enhancing the expressiveness, accuracy and generalization ability of the text summary generation model, and thus improving the overall performance of the text summary generation model, so that a more accurate and fluent text summary can be quickly generated through a higher-performance text summary generation model, and the quality and efficiency of text summary generation are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of the training method of the text summary generation model provided in the embodiment of the present application;

[0021] Figure 2 It is a schematic diagram of the architecture of the TTT-Linear-Transformer model provided in the embodiment of the present application;

[0022] Figure 3 yes Figure 1 A detailed flowchart of step S103 in FIG.

[0023] Figure 4 yes Figure 1 A detailed flowchart of step S104 in FIG.

[0024] Figure 5 is a schematic block diagram of a training device for a text summary generation model provided in an embodiment of the present application;

[0025] Figure 6 It is a schematic block diagram of the structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0029] First, some nouns involved in this application are analyzed:

[0030] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0031] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0032] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0033] Automatic Speech Recognition (ASR) technology: a technology that converts human speech signals into machine-readable text or commands.

[0034] TTT-Linear-Transformer model: is an improved Transformer model that uses a linear self-attention mechanism to replace the self-attention mechanism in the traditional Transformer model. The TTT-Linear-Transformer model aims to improve the efficiency and performance when processing long sequence data through test-time training (TTT). The main features and advantages of the TTT-Linear-Transformer model include the following aspects:

[0035] Linear complexity: Unlike the quadratic complexity of the traditional Transformer model, the computational complexity of the TTT-Linear-Transformer model grows linearly, which can reduce the amount of computation that depends on the sequence length, making it more efficient when processing large-scale data.

[0036] Long-distance dependency capture: When processing long text data, the TTT-Linear-Transformer model can capture long-distance dependencies more accurately, which is difficult for traditional Transformer models to achieve.

[0037] Stronger expressive power: The TTT-Linear-Transformer model unlocks the linear complexity architecture through expressive memory and has stronger expressive power than RNN.

[0038] Flexible adaptation to long context: The TTT-Linear-Transformer model performs well in long context, can dynamically adjust hidden states, has good generalization ability, and significantly improves efficiency and accuracy.

[0039] The TTT-Linear-Transformer model not only surpasses the traditional Transformer model in performance through its unique test-time training mechanism and linear complexity, but also shows significant advantages in efficiency and expressiveness.

[0040] Based on this, the embodiments of the present application provide a training method, apparatus, device and storage medium for a text summary generation model, aiming to improve the performance of the text summary generation model and improve the quality and efficiency of text summary generation.

[0041] The training method, apparatus, device and storage medium of the text summary generation model provided in the embodiments of the present application are specifically illustrated by the following embodiments. First, the training method of the text summary generation model in the embodiments of the present application is described.

[0042] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0043] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0044] The training method of the text summary generation model provided in the embodiment of the present application relates to the fields of financial technology and artificial intelligence technology. The training method of the text summary generation model provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application of the training method for the text summary generation model, etc., but is not limited to the above forms.

[0045] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0046] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0047] Figure 1 is an optional flowchart of the training method of the text summary generation model provided in the embodiment of the present application, Figure 1The method may include but is not limited to steps S101 to S104.

[0048] Step S101, obtaining a subword index sequence of a text sample.

[0049] Text samples may include financial text samples, which may be text data used in the field of financial technology, such as financial financial reports, financial meeting records, financial news articles, financial product introductions, etc. Text samples may also include medical text samples, which may be text data used in the medical field, such as case records, examination reports, clinical notes, etc. Text samples may also include other news material texts, etc.

[0050] The text samples are used to train the preset TTT-Linear-Transformer model, where there is no limit on the volume of the text samples.

[0051] By segmenting the text sample, a sub-word index sequence consisting of each sub-word (token) of the text sample can be obtained.

[0052] Step S102, processing the subword index sequence to obtain a coding tensor sequence.

[0053] In order to facilitate the understanding of the TTT-Linear-Transformer model, the subword index sequence of the text sample is processed to obtain the encoding tensor sequence.

[0054] Step S103, performing inner loop training and outer loop training on the preset TTT-Linear-Transformer model according to the encoded tensor sequence to obtain the target loss function of the TTT-Linear-Transformer model.

[0055] In order to enable the TTT-Linear-Transformer model to achieve efficient and effective summary generation learning, the encoded tensor sequence is used as the input of the TTT-Linear-Transformer model, self-supervised inner loop training is performed on the TTT layer, and the TTT-Linear-Transformer model is trained on the outer loop with the cooperation of the inner loop training, so as to obtain the target loss function of the TTT-Linear-Transformer model.

[0056] Among them, inner loop training is training at the micro level. Inner loop training focuses on the short-term quality of text summary generation. It can compress the context into a hidden state. This hidden state contains the context information of each subword in the text sample (that is, the semantics and position relationship of the subword in the text sample). Using the hidden state to generate output can make full and effective use of context information and ensure that the generated content is coherent with the previous content, which makes the generated summary logical and fluent.

[0057] Outer loop training is macro-level training, which optimizes the weights of the TTT-Linear-Transformer model from a global perspective, and can effectively improve the robustness and adaptability of the TTT-Linear-Transformer model.

[0058] Step S104: According to the target loss function, the weights of the TTT-Linear-Transformer model are updated in a dual form until the TTT-Linear-Transformer model converges to obtain a text summary generation model.

[0059] Considering that the TTT layer has only a few matrix multiplications (matmuls), in order to significantly reduce the consumption of computing resources and improve the training speed, the gradient of the input at each time step is not directly calculated. Instead, a dual form is introduced. Based on the target loss function, the gradient is implicitly calculated indirectly through matrix operations, thereby updating the weights of the TTT-Linear-Transformer model and obtaining the final updated weights.

[0060] It should be noted that the result of indirectly calculating the gradient for weight update through the dual form is the same as the result of directly calculating the gradient for weight update. It can greatly reduce the amount of calculation in the training process while ensuring the accuracy of the weight update.

[0061] When the updated weight is less than the preset weight threshold, the TTT-Linear-Transformer model is determined to have converged, and a trained TTT-Linear-Transformer model is obtained, and the trained TTT-Linear-Transformer model is used as a text summary generation model.

[0062] In this way, through the coordinated training of the inner loop and the outer loop, the TTT-Linear-Transformer model can handle short-term and long-term dependencies at the same time, so that it can efficiently and effectively learn to generate accurate and fluent summaries from the context, and by introducing the dual form, the TTT-Linear-Transformer model can more effectively utilize the input information, thereby reducing unnecessary calculations and improving training efficiency. As a result, the training effect is improved, which greatly enhances the expressiveness, accuracy and generalization ability of the text summary generation model, thereby improving the overall performance of the text summary generation model.

[0063] In some embodiments, in order to further improve the performance of the text summary generation model, the text summary generation model may be further optimized periodically according to actual needs.

[0064] The text summary generation model can be applied to text summary tasks. For example, in the field of financial technology, such as the summary generation scenario of financial live broadcasts, the text summary generation model can automatically and quickly generate high-quality summaries of financial live broadcasts.

[0065] Specifically, first obtain the financial live video data that needs to perform the text summarization task, where the financial live video data refers to the live video data in the field of financial technology, including video data of live introductions of financial products (such as property insurance), video data of live popularization of financial knowledge, etc.

[0066] The financial voice data contained in the financial live video data is then extracted, and the financial voice data is converted into text using ASR technology to obtain the target text.

[0067] The target text is segmented to obtain a target subword index sequence, and then the target subword index sequence is processed to obtain a target encoding tensor sequence.

[0068] The text summary generation model is used to perform summary generation processing on the target encoding tensor sequence to obtain a summary word vector sequence generated by the text summary generation model. Each word vector in the summary word vector sequence is converted into a summary word, and the summary of the target text can be output. In this way, high-quality summaries of financial live video data can be quickly generated in an end-to-end manner.

[0069] The training method of the text summary generation model provided in the above embodiment obtains a subword index sequence of a text sample; processes the subword index sequence to obtain a coding tensor sequence; performs inner loop training and outer loop training on a preset TTT-Linear-Transformer model according to the coding tensor sequence to obtain a target loss function of the TTT-Linear-Transformer model; according to the target loss function, updates the weights of the TTT-Linear-Transformer model in a dual form until the TTT-Linear-Transformer model converges to obtain a text summary generation model, which is used to generate a summary of the text. In this way, on the one hand, since the inner loop training provides local optimization in the summary generation process, the outer loop training ensures the global optimization of the TTT-Linear-Transformer model. Through the coordinated training of the inner loop and the outer loop, the TTT-Linear-Transformer model can handle short-term and long-term dependencies at the same time, so that it can efficiently and effectively learn to generate accurate and fluent summaries from the context, and realize the optimization training of the TTT-Linear-Transformer model for summary generation learning at the micro and macro levels; on the other hand, by introducing the dual form in the TTT-Linear-Transformer model, the TTT-Linear-Transformer model can more effectively utilize the information of the encoding tensor sequence, thereby reducing unnecessary calculations and improving training efficiency. As a result, the training effect is improved, thereby greatly enhancing the expressiveness, accuracy and generalization ability of the text summary generation model, and thus improving the overall performance of the text summary generation model, so that a more accurate and fluent text summary can be quickly generated through a higher-performance text summary generation model, and the quality and efficiency of text summary generation are improved.

[0070] In step S102 of some embodiments, the subword index sequence may be embedded to obtain an embedded vector sequence; the embedded vector sequence may be rotationally encoded to obtain an original encoding tensor sequence; and the original encoding tensor sequence may be normalized to obtain an encoding tensor sequence.

[0071] See also Figure 2 , Figure 2 The figure is a schematic diagram of the architecture of the TTT-Linear-Transformer model. The TTT-Linear-Transformer model mainly includes an embedding layer, a normalization layer, a TTT layer, a residual connection layer, and a linear transformation layer.

[0072] exist Figure 2 In the diagram, in order to accurately capture the semantic information of each subword in the subword index sequence, each subword in the subword index sequence can be embedded in the Embedding layer to obtain the embedding vector of each subword. The embedding vectors of all subwords constitute an embedding vector sequence.

[0073] In order to help the TTT-Linear-Transformer model understand the order of each subword, the embedding vector of each subword in the embedding vector sequence is rotationally encoded (Rotary position encoding) to obtain the original encoding tensor of each subword. The original encoding tensors of all subwords constitute the original encoding tensor sequence.

[0074] In order to improve the training speed and training stability, the original coding tensor of each subword in the original coding tensor sequence can be normalized (Layer Normalization) in the normalization layer to obtain the coding tensor of each subword, and the coding tensors of all subwords constitute the coding tensor sequence.

[0075] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S1031 to S1033.

[0076] Step S1031, performing inner loop training on the TTT layer of the TTT-Linear-Transformer model according to the encoded tensor sequence, obtaining a loss function, and fine-tuning the weights of the TTT-Linear-Transformer model according to the loss function to obtain fine-tuned weights, and generating a prediction output based on the fine-tuned weights;

[0077] Step S1032, constructing an identity function through residual learning;

[0078] Step S1033, performing outer loop training based on the predicted output, and constructing a target loss function through an identity function.

[0079] The encoded tensor sequence is input into the TTT layer of the TTT-Linear-Transformer model, and self-supervised inner loop training is performed in the TTT layer to obtain the loss function of the TTT-Linear-Transformer model.

[0080] In step S1031 of some embodiments, obtaining the loss function may be performing subspace projection processing on the encoded tensor sequence to obtain a training view, a label view, and a prediction view; and calculating the loss function based on the training view and the label view.

[0081] In the TTT layer, the θ K Matrix (key-value parameter matrix), θ V matrix (value parameter matrix) and θ Q Matrix (query parameter matrix), projects the encoded tensor sequence into the subspace to obtain the training view (train view), label view (label view) and prediction view (test view).

[0082] The training view, label view, and prediction view contain rich semantic information of text samples, which can help the TTT-Linear-Transformer model better understand the content and structure of text samples.

[0083] Among them, the training view is used to learn the main features of the encoded tensor sequence; the label view is used to provide self-supervision signals to help the TTT-Linear-Transformer model learn more accurate outputs; the prediction view is used to evaluate the generalization ability of the TTT-Linear-Transformer model to ensure that the TTT-Linear-Transformer model performs well on unseen data.

[0084] In some embodiments, subspace projection processing is performed on the coding tensor sequence to obtain the training view, label view and prediction view. The coding tensor sequence is dot-multiplied with the key parameter matrix of the TTT layer to obtain the training view; the coding tensor sequence is dot-multiplied with the value parameter matrix of the TTT layer to obtain the label view; the coding tensor sequence is dot-multiplied with the query parameter matrix of the TTT layer to obtain the prediction view.

[0085] Please continue reading Figure 2 , for the encoded tensor sequence X = [x1, ..., x b ], the input xt of time step t (t = 1, ..., b) is dot-multiplied with the θK matrix, θV matrix and θQ matrix of the TTT layer to obtain the training view θ K x t 、Tag viewθ V x t and the test view θ Q x t .

[0086] In this way, the loss function is calculated based on the training view and the label view.

[0087] It should be noted that the key idea of ​​the TTT-Linear-Transformer model is to use self-supervised learning to compress contextual information into a hidden state, regard the contextual information as unlabeled training data through the TTT layer, and regard the hidden state as a linear model (that is, the TTT layer performs an autoregressive mapping from one sequence to another, and the linear model can be understood as the one obtained by the TTT layer mapping). A large amount of training data is compressed into the weights of the TTT-Linear-Transformer model, so that its weights can effectively reflect and generalize the contextual information. In this way, self-supervised learning is used as the update rule to realize the weight update using the contextual information itself.

[0088] Among them, the linear model can be obtained by the linear function f lin (x)=Wx, then f lin The loss function l(W; x) t ) is as follows:

[0089] l(W;x t )=||f(θ K x t ; W)-θ V x t || 2

[0090] Among them, W is the weight of the TTT-Linear-Transformer model.

[0091] Next, in the inner loop training of the TTT layer, the weights of the TTT-Linear-Transformer model are fine-tuned according to the loss function to obtain the fine-tuned weights.

[0092] Exemplarily, the fine-tuning rule is batch gradient descent, and the weights after fine-tuning are as follows:

[0093]

[0094] Among them, W t represents the weight after fine-tuning;

[0095] W0 represents the initial weight of the TTT-Linear-Transformer model;

[0096] is the loss function l(W0;x t ) with respect to the initial weight W0;

[0097] η represents the learning rate of the TTT-Linear-Transformer model, and η can take the value of 1 / 2;

[0098] T represents the transpose operation, which is used to ensure the consistency of dimensions in matrix operations.

[0099] After obtaining the fine-tuned weights, you can generate predicted outputs based on the fine-tuned weights.

[0100] In step S1031 of some embodiments, generating the prediction output based on the fine-tuned weights may be generating the prediction output according to the prediction view and the fine-tuned weights.

[0101] According to the test view θ Q x t and the fine-tuned weights W t , get the predicted output z t , predict output z t As shown below:

[0102]

[0103] Among them, z t It is the prediction output generated by the TTT-Linear-Transformer model at time step t. It carries the key information of the current context and can reflect the understanding and prediction ability of the TTT-Linear-Transformer model at a specific time step.

[0104] In deep learning, the deeper the network layers, the better the model performance. When the number of network layers increases to a certain level, if the number of network layers is further increased, the accuracy of the model will decrease. Therefore, please continue to refer to Figure 2 Based on the inspiration of residual learning, a residual connection layer can be added to the architecture of the TTT-Linear-Transformer model to cleverly introduce the design of identity mapping in the TTT layer. Among them, identity mapping refers to the operation of directly passing the input to the output.

[0105] Specifically, in the residual connection layer, the encoded tensor sequence X = [x1, ..., x b ] and the linear model f mapped to the TTT layer network lin (x) is added to construct the identity function to ensure that the TTT-Linear-Transformer model can obtain more information in subsequent training. The identity function is as follows:

[0106] f(x)=x+LN(f lin (x)

[0107] In this way, introducing the design of identity mapping in the TTT layer can enhance the stability and expressiveness of the TTT-Linear-Transformer model.

[0108] Afterwards, the prediction output z obtained in the inner loop training is t The loss function is passed to the outer loop for outer loop training.

[0109] In the outer loop training, the output is constrained by the identity function, so let θ K =θ V =θ Q =I is the identity matrix, based on the predicted output z t The objective loss function is constructed as:

[0110] l(W;x t )=‖f(x t ; W)-x t ‖ 2 =‖W0x t -x t ‖ 2

[0111] In this way, based on the target loss function, the amount of computation required to calculate the final updated weights can be significantly reduced.

[0112] See also Figure 4 In some embodiments, step S104 includes but is not limited to steps S1041 to S1044.

[0113] Step S1041, according to the target loss function, implicitly obtain the gradient through the dual form;

[0114] Step S1042, updating the weights of the TTT-Linear-Transformer model according to the gradient to obtain updated weights;

[0115] Step S1043, when the updated weight is less than the preset weight threshold, determining that the TTT-Linear-Transformer model has converged, and obtaining a trained TTT-Linear-Transformer model;

[0116] Step S1044, using the trained TTT-Linear-Transformer model as a text summary generation model.

[0117] Specifically, through the dual form, matrix operations are used to implicitly calculate the gradient, the gradient G t As shown below:

[0118]

[0119] According to the gradient G t , update the weight, and finally the updated weight W b can be simplified to:

[0120]

[0121] Therefore, it is more convenient to calculate W by matrix multiplication b .

[0122] In order to calculate the final output token Z = [z1, ..., z b ],because

[0123]

[0124] Therefore, a mask matrix can be introduced, and the mask matrix Δ=[δ1,...,δ b ],in The mask matrix Δ can be derived as:

[0125] Δ=mask(X T X)(W0X-X)

[0126] Mask Matrix The mask matrix Δ is a lower triangular matrix with all elements above the diagonal being zero (similar to the attention mask), which ensures dependencies in the calculation. The mask matrix Δ can also be conveniently calculated by matrix multiplication.

[0127] Thus, the term W0X-X can be used in the calculation of W b and z t Reuse in the process:

[0128] W b =W0-2η(W0X-X)X T

[0129] z t =W0x t -2η·mask(X T X)(W0X-X)

[0130] It can be seen that the final weight update W is calculated b and output token z t In the process of calculating W0X-X, the expression of calculating W0X-X is involved. b It is used when calculating the mask matrix Δ, so repeated calculations can be avoided and the previously calculated results can be directly used to speed up the calculation.

[0131] In this way, by introducing residual learning, dual form and mask matrix into the TTT-Linear-Transformer model, the TTT-Linear-Transformer model can utilize the input information more effectively, which not only improves the computational efficiency but also enhances the learning ability of the TTT-Linear-Transformer model, thereby improving the training effect.

[0132] See also Figure 5 , Figure 5 It is a schematic block diagram of a training device for a text summary generation model provided in an embodiment of the present application. The training device for the text summary generation model can be configured in a computer device to execute the aforementioned training method for the text summary generation model.

[0133] like Figure 5 As shown, the training device 500 for the text summary generation model includes: a sample acquisition module 501, an encoding processing module, a model training module 503 and a weight updating module 504.

[0134] A sample acquisition module 501 is used to acquire a subword index sequence of a text sample;

[0135] An encoding processing module 502 is used to process the subword index sequence to obtain an encoding tensor sequence;

[0136] A model training module 503 is used to perform inner loop training and outer loop training on a preset TTT-Linear-Transformer model according to the encoded tensor sequence to obtain a target loss function of the TTT-Linear-Transformer model;

[0137] The weight updating module 504 is used to update the weights of the TTT-Linear-Transformer model in a dual form according to the target loss function until the TTT-Linear-Transformer model converges to obtain a text summary generation model, and the text summary generation model is used to generate a summary of the text.

[0138] In one embodiment, the model training module 503 is further used to:

[0139] According to the encoded tensor sequence, inner loop training is performed on the TTT layer of the TTT-Linear-Transformer model to obtain a loss function, and weights of the TTT-Linear-Transformer model are fine-tuned according to the loss function to obtain fine-tuned weights, and prediction outputs are generated based on the fine-tuned weights;

[0140] Constructing identity functions through residual learning;

[0141] Outer loop training is performed based on the predicted output, and the target loss function is constructed through the identity function.

[0142] In one embodiment, the model training module 503 is further used to:

[0143] Performing subspace projection processing on the encoded tensor sequence to obtain a training view, a label view, and a prediction view;

[0144] The loss function is calculated according to the training view and the label view.

[0145] In one embodiment, the model training module 503 is further used to:

[0146] Performing a dot multiplication of the encoded tensor sequence and the key-value parameter matrix of the TTT layer to obtain the training view;

[0147] Performing a dot multiplication of the encoding tensor sequence and the value parameter matrix of the TTT layer to obtain the label view;

[0148] The encoded tensor sequence is dot-multiplied with the query parameter matrix of the TTT layer to obtain the predicted view.

[0149] In one embodiment, the model training module 503 is further used to:

[0150] The prediction output is generated according to the prediction view and the fine-tuned weight.

[0151] In one embodiment, the weight updating module 504 is further used to:

[0152] According to the objective loss function, the gradient is implicitly obtained through the dual form;

[0153] According to the gradient, updating the weights of the TTT-Linear-Transformer model to obtain updated weights;

[0154] When the updated weight is less than a preset weight threshold, determining that the TTT-Linear-Transformer model converges, and obtaining a trained TTT-Linear-Transformer model;

[0155] The trained TTT-Linear-Transformer model is used as the text summary generation model.

[0156] In one embodiment, the encoding processing module 502 is further used to:

[0157] Embedding the subword index sequence to obtain an embedding vector sequence;

[0158] Performing rotation position encoding on the embedding vector sequence to obtain an original encoding tensor sequence;

[0159] The original coding tensor sequence is normalized to obtain a coding tensor sequence.

[0160] Among them, each module in the above-mentioned training device 500 for the text summary generation model corresponds to each step in the above-mentioned training method embodiment for the text summary generation model, and their functions and implementation processes are no longer described one by one here.

[0161] The training device 500 for the text summary generation model can execute the training method for the text summary generation model provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved by the training method for the text summary generation model provided in the embodiment of the present application can be achieved. Please refer to the previous embodiment for details and will not be repeated here.

[0162] The method and apparatus of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0163] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 6 Runs on the computer device shown.

[0164] See also Figure 6 , Figure 6 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0165] See also Figure 6 The computer device includes a processor and a memory connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0166] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0167] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any training method for the text summary generation model.

[0168] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0169] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0170] Get the subword index sequence of the text sample;

[0171] Processing the subword index sequence to obtain an encoding tensor sequence;

[0172] Performing inner loop training and outer loop training on a preset TTT-Linear-Transformer model according to the encoded tensor sequence to obtain a target loss function of the TTT-Linear-Transformer model;

[0173] According to the target loss function, the weights of the TTT-Linear-Transformer model are updated in a dual form until the TTT-Linear-Transformer model converges, thereby obtaining a text summary generation model, which is used to generate a summary of the text.

[0174] In one embodiment, when the processor implements the inner loop training and the outer loop training of the preset TTT-Linear-Transformer model according to the encoded tensor sequence to obtain the target loss function of the TTT-Linear-Transformer model, it is used to implement:

[0175] According to the encoded tensor sequence, inner loop training is performed on the TTT layer of the TTT-Linear-Transformer model to obtain a loss function, and weights of the TTT-Linear-Transformer model are fine-tuned according to the loss function to obtain fine-tuned weights, and prediction outputs are generated based on the fine-tuned weights;

[0176] Constructing identity functions through residual learning;

[0177] Outer loop training is performed based on the predicted output, and the target loss function is constructed through the identity function.

[0178] In one embodiment, when implementing the acquisition of the loss function, the processor is used to implement:

[0179] Performing subspace projection processing on the encoded tensor sequence to obtain a training view, a label view, and a prediction view;

[0180] The loss function is calculated according to the training view and the label view.

[0181] In one embodiment, when the processor performs subspace projection processing on the encoded tensor sequence to obtain a training view, a label view, and a prediction view, it is configured to implement:

[0182] Performing a dot multiplication of the encoded tensor sequence and the key-value parameter matrix of the TTT layer to obtain the training view;

[0183] Performing a dot multiplication of the encoding tensor sequence and the value parameter matrix of the TTT layer to obtain the label view;

[0184] The encoded tensor sequence is dot-multiplied with the query parameter matrix of the TTT layer to obtain the predicted view.

[0185] In one embodiment, when the processor generates the prediction output based on the fine-tuned weight, it is used to implement:

[0186] The prediction output is generated according to the prediction view and the fine-tuned weight.

[0187] In one embodiment, when the processor implements updating the weights of the TTT-Linear-Transformer model in a dual form according to the target loss function until the TTT-Linear-Transformer model converges to obtain a text summary generation model, it is used to implement:

[0188] According to the objective loss function, the gradient is implicitly obtained through the dual form;

[0189] According to the gradient, updating the weights of the TTT-Linear-Transformer model to obtain updated weights;

[0190] When the updated weight is less than a preset weight threshold, determining that the TTT-Linear-Transformer model converges, and obtaining a trained TTT-Linear-Transformer model;

[0191] The trained TTT-Linear-Transformer model is used as the text summary generation model.

[0192] In one embodiment, when the processor processes the subword index sequence to obtain the encoding tensor sequence, it is used to implement:

[0193] Embedding the subword index sequence to obtain an embedding vector sequence;

[0194] Performing rotation position encoding on the embedding vector sequence to obtain an original encoding tensor sequence;

[0195] The original coding tensor sequence is normalized to obtain a coding tensor sequence.

[0196] The computer device can execute the training method of the text summary generation model provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved by the training method of the text summary generation model provided in the embodiment of the present application can be achieved. Please refer to the previous embodiment for details and will not be repeated here.

[0197] An embodiment of the present application also provides a computer-readable storage medium.

[0198] The computer-readable storage medium of the present application stores a computer program, and when the computer program is executed by a processor, the steps of the training method of the text summary generation model as described above are implemented.

[0199] The computer-readable storage medium may be an internal storage unit of the training device or computer device of the text summary generation model described in the above embodiments, such as a hard disk or memory of the training device or computer device of the text summary generation model. The computer-readable storage medium may also be an external storage device of the training device or computer device of the text summary generation model, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital Card, SD Card), a flash card (Flash Card), etc. equipped on the training device or computer device of the text summary generation model.

[0200] Since the computer program stored in the computer-readable storage medium can execute any of the training methods for the text summary generation model provided in the embodiments of the present application, the beneficial effects that can be achieved by any of the training methods for the text summary generation model provided in the embodiments of the present application can be achieved. For details, please refer to the previous embodiments and will not be repeated here.

[0201] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0202] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0203] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0204] The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and these modifications or substitutions should be included in the protection scope of the present application.

Claims

1. A training method for a text summary generation model, characterized in that: The method comprises: Get the subword index sequence of the text sample; Processing the subword index sequence to obtain an encoding tensor sequence; Performing inner loop training and outer loop training on a preset TTT-Linear-Transformer model according to the encoded tensor sequence to obtain a target loss function of the TTT-Linear-Transformer model; According to the target loss function, the weights of the TTT-Linear-Transformer model are updated in a dual form until the TTT-Linear-Transformer model converges, thereby obtaining a text summary generation model, which is used to generate a summary of the text.

2. The method according to claim 1, characterized in that The inner loop training and the outer loop training are performed on the preset TTT-Linear-Transformer model according to the encoded tensor sequence to obtain the target loss function of the TTT-Linear-Transformer model, including: According to the encoded tensor sequence, inner loop training is performed on the TTT layer of the TTT-Linear-Transformer model to obtain a loss function, and weights of the TTT-Linear-Transformer model are fine-tuned according to the loss function to obtain fine-tuned weights, and prediction outputs are generated based on the fine-tuned weights; Constructing identity functions through residual learning; Outer loop training is performed based on the predicted output, and the target loss function is constructed through the identity function.

3. The method according to claim 2, characterized in that The obtaining of the loss function comprises: Performing subspace projection processing on the encoded tensor sequence to obtain a training view, a label view, and a prediction view; The loss function is calculated according to the training view and the label view.

4. The method according to claim 3, characterized in that The performing subspace projection processing on the encoded tensor sequence to obtain a training view, a label view and a prediction view includes: Performing a dot multiplication of the encoded tensor sequence and the key-value parameter matrix of the TTT layer to obtain the training view; Performing a dot multiplication of the encoding tensor sequence and the value parameter matrix of the TTT layer to obtain the label view; The encoded tensor sequence is dot-multiplied with the query parameter matrix of the TTT layer to obtain the predicted view.

5. The method according to claim 3, characterized in that: Generating a prediction output based on the fine-tuned weights includes: The prediction output is generated according to the prediction view and the fine-tuned weight.

6. The method according to any one of claims 1 to 5, characterized in that: The method of updating the weight of the TTT-Linear-Transformer model in a dual form according to the target loss function until the TTT-Linear-Transformer model converges to obtain a text summary generation model includes: According to the target loss function, the gradient is implicitly obtained through the dual form; According to the gradient, updating the weights of the TTT-Linear-Transformer model to obtain updated weights; When the updated weight is less than a preset weight threshold, determining that the TTT-Linear-Transformer model converges, and obtaining a trained TTT-Linear-Transformer model; The trained TTT-Linear-Transformer model is used as the text summary generation model.

7. The method according to any one of claims 1 to 5, characterized in that: The processing of the subword index sequence to obtain a coding tensor sequence includes: Embedding the subword index sequence to obtain an embedding vector sequence; Performing rotation position encoding on the embedding vector sequence to obtain an original encoding tensor sequence; The original coding tensor sequence is normalized to obtain a coding tensor sequence.

8. A training device for a text summary generation model, characterized in that: The device comprises: A sample acquisition module is used to obtain a subword index sequence of a text sample; An encoding processing module, used for processing the subword index sequence to obtain an encoding tensor sequence; A model training module, used to perform inner loop training and outer loop training on a preset TTT-Linear-Transformer model according to the encoded tensor sequence, and obtain a target loss function of the TTT-Linear-Transformer model; A weight updating module is used to update the weights of the TTT-Linear-Transformer model in a dual form according to the target loss function until the TTT-Linear-Transformer model converges to obtain a text summary generation model, and the text summary generation model is used to generate a summary of the text.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the training method of the text summary generation model described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the training method of the text summary generation model described in any one of claims 1 to 7 is implemented.

Images