Data processing method and device, abstract generation method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a data processing method and device, equipment, a storage medium and a product. The method comprises the steps of obtaining a text task training data set and an abstract training data set, training N to-be-trained matrixes deployed by a first to-be-trained model through the text task training data set to obtain N auxiliary task matrixes, and calling a second to-be-trained model to perform abstract generation processing on a sample text, and obtaining an abstract generation result of the sample text, deploying the N auxiliary task matrixes on a bypass of a gating full connection layer of a second to-be-trained model, and adjusting an expert matrix deployed in the second to-be-trained model based on a difference between the abstract generation result and a target abstract to obtain an abstract generation model. Therefore, in the data processing process, the decoupling auxiliary task matrix and the expert matrix are respectively trained, and the to-be-trained model can be subjected to targeted training from multiple dimensions, so that the accuracy of the text abstract generated by the abstract generation model is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data processing method, a summary generation method, a data processing apparatus, a summary generation apparatus, a computer device, a computer-readable storage medium, a data processing product, and a summary generation product. Background Technology

[0002] With advancements in scientific research, technologies related to artificial intelligence (AI) models have developed rapidly. AI models are widely used in fields such as code generation, natural language question answering, and summarization. In the field of summarization, AI models (summarization models) can generate text summaries based on input text, enabling people to efficiently extract the main content of the text. Research has found that if only the summarization task itself is focused on during the training process of AI models, the text summaries generated by the trained summarization models are often not accurate enough. Summary of the Invention

[0003] This application provides a data processing method, a summary generation method, an apparatus, a device, and a computer-readable storage medium, which can improve the accuracy of text summaries generated by the summary generation model.

[0004] On one hand, embodiments of this application provide a data processing method, including: Obtain the text task training dataset and the summary training dataset. The text task training dataset includes N auxiliary task training datasets. The summary training dataset includes sample text and target summary. The target summary is the labeled data of the sample text. N is an integer greater than 1. The first model to be trained is trained using a text task training dataset to obtain N auxiliary task matrices. The first model to be trained includes a first feedforward neural network layer, and the sidebar of the first feedforward neural network layer has N training matrices. The i-th auxiliary task matrix is ​​obtained by training the i-th auxiliary task matrix using the i-th auxiliary task training dataset, where i is a positive integer less than or equal to N. The second model to be trained is called to perform a summary generation process on the sample text, and the summary generation result of the sample text is obtained. The second model to be trained includes a gated fully connected layer and a second feedforward neural network layer. N auxiliary task matrices are deployed in the bypass of the gated fully connected layer, and expert matrices are deployed in the bypass of the second feedforward neural network layer. Based on the difference between the generated summary and the target summary, the expert matrix is ​​adjusted to obtain the summary generation model, which is used to generate text summaries.

[0005] In this embodiment, a text task training dataset and a summary training dataset are obtained. The first model to be trained is trained using the text task training dataset to obtain N auxiliary task matrices. The first feedforward neural network layer of the first model to be trained has N training matrices deployed in its sidepath. These N auxiliary task matrices are obtained by training the N training matrices using the text task training dataset. A second model to be trained is then called to perform summary generation processing on sample text in the summary training dataset to obtain the summary generation result of the sample text. The N auxiliary task matrices are deployed in the sidepath of the gated fully connected layer of the second model to be trained, and an expert matrix is ​​deployed in the sidepath of the second feedforward neural network layer of the second model to be trained. Based on the difference between the summary generation result and the target summary, the expert matrix is ​​adjusted to obtain the summary generation model. It can be seen that by adding training auxiliary task matrices and decoupling the training processes of the auxiliary task matrices and expert matrices during data processing, the model to be trained can be targeted from multiple dimensions, thereby improving the accuracy of the text summaries generated by the summary generation model.

[0006] On the one hand, embodiments of this application provide a method for generating an abstract, including: Obtain the intermediate sequence corresponding to the text to be processed. The intermediate sequence is obtained by encoding the relationship between different text positions in the text to be processed. The results of at least two auxiliary tasks are weighted and fused to obtain the weighted result corresponding to the intermediate sequence. The results of different auxiliary tasks are obtained by performing feature transformation on the intermediate sequence using different auxiliary task matrices. The weights corresponding to each auxiliary task result are determined based on the intermediate sequence. A nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed. A summary of the text to be processed is generated based on the feature representation sequence of the text to be processed.

[0007] In this embodiment, an intermediate sequence corresponding to the text to be processed is obtained. This intermediate sequence is obtained by encoding the associations between different text positions within the text. The results of at least two auxiliary tasks are weighted and fused to obtain a weighted result corresponding to the intermediate sequence. Different auxiliary task results are obtained by performing feature transformations on the intermediate sequence using different auxiliary task matrices. The weights corresponding to each auxiliary task result are determined based on the intermediate sequence. A nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain a feature representation sequence of the text to be processed. Based on this feature representation sequence, a summary of the text to be processed is generated. Therefore, by weightedly fusing the results of multiple auxiliary tasks, the model can gain a more comprehensive and profound understanding of the text to be processed during the summary generation process, thereby improving the accuracy of the text summary generated by the summary generation model.

[0008] On one hand, embodiments of this application provide a data processing apparatus, which includes: The acquisition unit is used to acquire the text task training dataset and the summary training dataset. The text task training dataset includes N auxiliary task training datasets, and the summary training dataset includes sample text and target summary. The target summary is the labeled data of the sample text, and N is an integer greater than 1. The processing unit is used to train the first model to be trained using a text task training dataset to obtain N auxiliary task matrices. The first model to be trained includes a first feedforward neural network layer, and the sidebar of the first feedforward neural network layer has N training matrices. The i-th auxiliary task matrix is ​​obtained by training the i-th training matrix using the i-th auxiliary task training dataset, where i is a positive integer less than or equal to N. And it is used to call the second model to be trained to perform summary generation processing on the sample text and obtain the summary generation result of the sample text; the second model to be trained includes a gated fully connected layer and a second feedforward neural network layer, N auxiliary task matrices are deployed in the bypass of the gated fully connected layer, and an expert matrix is ​​deployed in the bypass of the second feedforward neural network layer; And it is used to adjust the expert matrix based on the difference between the summary generation result and the target summary to obtain the summary generation model, which is used to generate text summaries.

[0009] In one implementation, the i-th auxiliary task training dataset includes the i-th training text and the labeled data of the i-th training text; the processing unit is used to train the first model to be trained using the text task training dataset to obtain N auxiliary task matrices, specifically for: Obtain N auxiliary task processing results from the output of N training matrices. The i-th auxiliary task processing result is obtained based on the feature representation sequence of the i-th training text. The feature representation sequence of the i-th training text is obtained by performing a linear transformation on the i-th training text based on the i-th training matrix. The prediction biases of the N training matrices are summed to obtain the first loss function; the prediction bias of the i-th training matrix is ​​obtained based on the difference between the processing result of the i-th auxiliary task and the labeled data of the i-th training text. With the goal of minimizing the first loss function, the N training matrices are adjusted to obtain N auxiliary task matrices.

[0010] In one implementation, the first feedforward neural network layer includes a first fully connected projection matrix; the processing unit is configured to obtain N auxiliary task processing results output by N matrices to be trained, specifically for: Obtain the input sequence of the first feedforward neural network layer. The input sequence is obtained by encoding the association between any two text positions in the i-th training text. The input sequence is nonlinearly transformed using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text. Based on the feature representation sequence of the i-th training text, generate the i-th auxiliary task processing result.

[0011] In one implementation, the first model to be trained further includes an attention layer, the sidepath of which is configured with a general matrix; the processing unit is used to acquire the input sequence of the first feedforward neural network layer, specifically for: The attention layer is invoked to encode the association between any two text positions in the i-th training text, thus obtaining the encoding result of the i-th training text; The first transformation result is obtained by performing a linear transformation on the i-th training text using a general matrix. The encoding result is enhanced by nonlinear mapping using the first transformation result to obtain the input sequence of the first feedforward neural network layer.

[0012] In one implementation, the processing unit is configured to perform a nonlinear transformation on the input sequence using a first fully connected projection matrix and the i-th training matrix to obtain a feature representation sequence of the i-th training text, specifically configured to: The input sequence is nonlinearly transformed using the first fully connected projection matrix to obtain the second transformation result; The i-th matrix to be trained is used to perform a linear transformation on the input sequence to obtain the third transformation result; The second transformation result is enhanced by nonlinear mapping using the third transformation result to obtain the feature representation sequence of the i-th training text.

[0013] In one implementation, the first model to be trained further includes an attention layer, the sidepath of which is configured with a general matrix; the processing unit is used to adjust the N training matrices to obtain N auxiliary task matrices with the objective of minimizing a first loss function, specifically for: With the goal of minimizing the first loss function, the general matrix and N training matrices are adjusted to obtain the adjusted general matrix and N auxiliary task matrices.

[0014] In one implementation, the first feedforward neural network layer includes a first fully connected projection matrix; the i-th auxiliary task training dataset includes the i-th training text and the labeled data of the i-th training text; the processing unit is used to train the first model to be trained using the text task training dataset to obtain N auxiliary task matrices, specifically for: Obtain the input sequence of the first feedforward neural network layer. The input sequence is obtained by encoding the association between any two text positions in the i-th training text. The input sequence is nonlinearly transformed using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text. Based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, the i-th training matrix is ​​adjusted to obtain the i-th auxiliary task matrix; the auxiliary task processing result of the i-th training text is generated based on the feature representation sequence of the i-th training text.

[0015] In one implementation, the first model to be trained further includes an attention layer, the sidepath of which is configured with a general matrix; the processing unit is used to adjust the i-th matrix to be trained based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, to obtain the i-th auxiliary task matrix, specifically for: Based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, the general matrix and the i-th training matrix are adjusted to obtain the adjusted general matrix and the i-th auxiliary task matrix.

[0016] In one implementation, the second model to be trained further includes an attention layer; the processing unit is used to call the second model to be trained to perform summary generation processing on the sample text, and obtain the summary generation result of the sample text, specifically for: By encoding the associations between different text positions in the sample text through an attention layer, the intermediate sequence corresponding to the sample text is obtained; The weighted result of the sample text is obtained by weighted fusion of the processing results of N auxiliary tasks through a gated fully connected layer; the processing results of N auxiliary tasks are obtained by feature transformation of the intermediate sequence using N auxiliary task matrices. The second feedforward neural network layer is invoked to perform a nonlinear transformation on the weighted result corresponding to the sample text, thereby obtaining the feature representation sequence of the sample text. Based on the feature representation sequence of the sample text, a summary of the sample text is generated.

[0017] In one implementation, the gated fully connected layer includes a second fully connected projection matrix; the processing unit is used to adjust the expert matrix based on the difference between the summary generation result and the target summary to obtain a summary generation model, specifically for: A second loss function is constructed based on the difference between the generated summary and the target summary. With the goal of minimizing the second loss function, the expert matrix and the second fully connected projection matrix are adjusted to obtain the summary generation model, which includes the adjusted expert matrix and the adjusted second fully connected projection matrix.

[0018] In one implementation, the processing unit is configured to acquire a text task training dataset, specifically for: Obtain N training texts and their labeled data; The N training texts and their labeled data are standardized according to the instruction set template to obtain N auxiliary task training datasets. The instruction set template includes a task instruction field, an input field, and a verification field; the value of the task instruction field of the i-th auxiliary task training dataset is generated based on the relationship between the i-th training text and the labeled data of the i-th training text.

[0019] In one implementation, the processing unit is configured to acquire a summary training dataset, specifically for: Get M related sub-texts and concatenate them to obtain the sample text, where M is an integer greater than 1; Obtain the labeled data of the sample text; The sample text and labeled data are standardized according to the instruction set template to obtain the summary training dataset. The instruction set template includes a task instruction field, an input field, and a validation field; the value of the task instruction field in the summary training dataset is used to describe the summary generation conditions.

[0020] On one hand, embodiments of this application provide a summary generation apparatus, which includes: The acquisition unit is used to acquire the intermediate sequence corresponding to the text to be processed. The intermediate sequence is obtained by encoding the relationship between different text positions in the text to be processed. The processing unit is used to perform weighted fusion of the processing results of at least two auxiliary tasks to obtain the weighted result corresponding to the intermediate sequence; the processing results of different auxiliary tasks are obtained by performing feature transformation on the intermediate sequence using different auxiliary task matrices, and the weights corresponding to each processing result of auxiliary tasks are determined based on the intermediate sequence; And a nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed; And a summary of the text to be processed, based on the feature representation sequence of the text to be processed.

[0021] In one implementation, the intermediate sequence is obtained by encoding the association between different text positions in the text to be processed through the attention layer in the summary generation model; the summary generation model also includes a gated fully connected layer, which includes a first fully connected projection matrix and at least two auxiliary task matrices, wherein the at least two auxiliary task matrices are bypass matrices of the first fully connected projection matrix. The processing unit is used to perform weighted fusion of the processing results of at least two auxiliary tasks to obtain a weighted result corresponding to the intermediate sequence, specifically for: The intermediate sequence is projected onto the gate space using the first fully connected projection matrix to obtain the weights corresponding to each auxiliary task matrix; Call at least two auxiliary task matrices to perform feature transformation on the intermediate sequence respectively, and obtain at least two auxiliary task processing results; The processing results of the corresponding auxiliary tasks are weighted and fused according to the weights of each auxiliary task matrix to obtain the weighted result corresponding to the intermediate sequence.

[0022] In one implementation, the intermediate sequence is obtained by encoding the associations between different text positions in the text to be processed through the attention layer in the summarization generation model; the summarization generation model further includes a feedforward neural network layer, which includes a second fully connected projection matrix and an expert matrix, wherein the expert matrix is ​​a bypass matrix of the second fully connected projection matrix; the processing unit is used to perform a nonlinear transformation on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed, specifically for: The first transformation result is obtained by performing a nonlinear transformation on the weighted result corresponding to the intermediate sequence using the second fully connected projection matrix. The second transformation result is obtained by linearly transforming the weighted result corresponding to the intermediate sequence using the expert matrix; The first transformation result is enhanced by nonlinear mapping using the second transformation result to obtain the feature representation sequence of the text to be processed.

[0023] In one implementation, the processing unit is configured to obtain an intermediate sequence corresponding to the text to be processed, specifically for: Obtain the text to be processed and divide it into N sub-sequence blocks according to a preset length, where N is an integer greater than 1; Based on the association between different text positions in each sub-sequence block, N sub-sequence blocks are encoded to obtain the encoding results of N sub-sequence blocks; the encoding result of any sub-sequence block is obtained by encoding the association between any two text positions in that sub-sequence block. The encoding results of N sub-sequence blocks are passed across blocks using a sliding window to obtain the information transmission result of the text to be processed. The encoding results of N sub-sequence blocks and the information transmission results of the text to be processed are aggregated to obtain the intermediate sequence corresponding to the text to be processed.

[0024] Accordingly, this application provides a computer device comprising: Memory, which stores computer programs; A processor is used to load a computer program to implement the above-described data processing method; or, to load a computer program to implement the above-described summary generation method.

[0025] Accordingly, this application provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described data processing method; or, adapted to be loaded by a processor and executed by the above-described digest generation method.

[0026] Accordingly, this application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned data processing method; or, to perform the aforementioned digest generation method.

[0027] In the data processing, a text task training dataset and a summary training dataset are acquired. The first model to be trained is trained using the text task training dataset to obtain N auxiliary task matrices. The side path of the first feedforward neural network layer of the first model to be trained has N training matrices. The N auxiliary task matrices are obtained by training the N training matrices using the text task training dataset. The second model to be trained is then called to perform summary generation processing on the sample text in the summary training dataset to obtain the summary generation result of the sample text. The N auxiliary task matrices are deployed in the side path of the gated fully connected layer of the second model to be trained, and the side path of the second feedforward neural network layer of the second model to be trained has an expert matrix. Based on the difference between the summary generation result and the target summary, the expert matrix is ​​adjusted to obtain the summary generation model. In the summary generation process, an intermediate sequence corresponding to the text to be processed is obtained. This intermediate sequence is obtained by encoding the associations between different text positions in the text. The results of at least two auxiliary tasks are weighted and fused to obtain a weighted result corresponding to the intermediate sequence. Different auxiliary task results are obtained by performing feature transformations on the intermediate sequence using different auxiliary task matrices. The weights corresponding to each auxiliary task result are determined based on the intermediate sequence. A nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed. Based on the feature representation sequence of the text to be processed, a summary of the text is generated. It is evident that by incorporating training auxiliary task matrices and decoupling the training processes of the auxiliary task matrices and expert matrices during data processing, the model to be trained can be targeted from multiple dimensions, thereby improving the accuracy of the text summarization generated by the summary generation model. By weightedly fusing the results of multiple auxiliary tasks during the summary generation process, the model can gain a more comprehensive and profound understanding of the text to be processed during the summary generation process, thus improving the accuracy of the text summarization generated by the summary generation model. Attached Figure Description

[0028] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A schematic diagram of a system architecture for applying the technical solutions of this application is provided as an embodiment of this application; Figure 2 An abstract generation architecture diagram provided for embodiments of this application; Figure 3 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 4 A schematic diagram of the architecture of a first model to be trained provided in an embodiment of this application; Figure 5 A schematic diagram of the architecture of a second model to be trained provided in an embodiment of this application; Figure 6 A flowchart illustrating a method for generating an abstract, as provided in an embodiment of this application; Figure 7 A flowchart illustrating another abstract generation method provided in this application embodiment; Figure 8 A schematic diagram of abstract generation provided for an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of an abstract generation device provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0030] It should be noted in advance that, in order to enable those skilled in the art to better understand the technical solutions proposed in the embodiments of this application, the embodiments of this application will be described clearly and completely in conjunction with one or more accompanying drawings. Furthermore, the various drawings shown in the embodiments of this application are merely illustrative examples; for example, the execution order of each step in the drawings can be adaptively adjusted according to the actual application scenario. In addition, in the embodiments of this application, the block diagrams shown in the various drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0033] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0034] It should be noted that "multiple" in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0035] This application provides a data processing scheme and a summary generation scheme, which can improve the accuracy of text summaries generated by the summary generation model. Figure 1 A schematic diagram of a system architecture for applying the technical solutions of this application is provided as an embodiment of this application, such as... Figure 1 As shown, the system architecture provided in this application includes a terminal device 101 and a server 102. The data processing scheme and summary generation scheme provided in this application can be executed by the terminal device 101 or the server 102, or they can be executed collaboratively by the terminal device 101 and the server 102. The terminal device may include, but is not limited to: smartphones (such as Android phones, iOS phones, HarmonyOS phones, etc.), tablet computers, desktop computers, portable personal computers, mobile internet devices (Mobile Internet Devices, MIDs), smart interactive devices, smart home appliances, vehicle terminals, wearable devices, etc., and this application embodiment does not limit this; the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and this application embodiment does not limit this.

[0036] It should be noted that, Figure 1The number of terminal devices and servers is for illustrative purposes only and does not constitute an actual limitation of this application. Terminal device 101 and server 102 can be connected via wired or wireless means, and this application does not impose any limitations on this. Furthermore, the text data involved in this application can be obtained by converting multimedia data such as audio, images, and video, and this application does not impose any limitations on this.

[0037] Figure 2 An abstract generation architecture diagram is provided for embodiments of this application, such as... Figure 2 As shown, after obtaining the text task training dataset and the summary training dataset, server 102 can first train the N training matrices deployed in the first training model using the text task training dataset to obtain N auxiliary task matrices, where N is an integer greater than 1. The N training matrices are deployed as bypasses of the first feedforward neural network layer in the first training model. Different auxiliary task matrices are used to process different text tasks, all of which are related to summary generation; for example, auxiliary task matrix 1 is used to process the keyword extraction task, auxiliary task matrix 2 is used to process the sentence order prediction task, auxiliary task matrix 3 is used to process the local summary generation task… auxiliary task matrix N is used to process the document importance scoring task.

[0038] In one embodiment, this application uses low-rank adaptation (LoRA) to fine-tune the parameters of the model to be trained. Low-rank adaptation is a type of efficient parameter fine-tuning (PEFT) technique that achieves efficient learning by freezing the weights of the pre-trained language model (i.e., the model to be trained) and adjusting only a limited number of parameters within several sets of low-rank matrices A and B added to the sidepaths of each network layer. In this case, any matrix to be trained in this application consists of low-rank matrix A and low-rank matrix B; assuming the original path of any network layer in the model is h = Wx; where W is the weight matrix of the network layer, x is the input data of the network layer, and h is the weighted result of the network layer on the input data (i.e., the output data). After deploying a matrix to be trained in the sidepath of the network layer, the path of the network layer is updated to h = Wx + ABx, where the dimension of AB is the same as that of W. Adjusting the matrix to be trained is equivalent to adjusting the low-rank matrices A and B, and the resulting complex task matrix consists of the adjusted low-rank matrices A and B. The general matrix and expert matrix involved in this application are similar to the matrix to be trained.

[0039] Next, server 102 can deploy N auxiliary task matrices to the second model to be trained, and call the second model to be trained with N auxiliary task matrices to perform summary generation processing on the sample text in the summary training dataset, and obtain the summary generation result of the sample text. The second model to be trained includes a gated fully connected layer and a second feedforward neural network layer. The N auxiliary task matrices are deployed in the bypass of the gated fully connected layer, and the bypass of the second feedforward neural network layer has an expert matrix deployed.

[0040] Next, server 102 adjusts the expert matrix based on the difference between the generated summary and the target summary in the training dataset to obtain the summary generation model. The summary generation model can be deployed on server 102 or on terminal device 101, and this application does not impose any restrictions on this.

[0041] In one embodiment, after acquiring the text to be processed, server 102 encodes the correlations between different text positions in the text to obtain an intermediate sequence corresponding to the text to be processed. Then, a gated fully connected layer with N auxiliary task matrices is deployed in a bypass manner to perform weighted fusion of the processing results of the N auxiliary tasks, resulting in a weighted result corresponding to the intermediate sequence. The processing result of the i-th auxiliary task is obtained by performing feature transformation on the intermediate sequence corresponding to the i-th auxiliary task matrix. After obtaining the weighted result corresponding to the intermediate sequence, a second feedforward neural network layer with an adjusted expert matrix is ​​deployed in a bypass manner to perform a nonlinear transformation on the weighted result corresponding to the intermediate sequence, obtaining a feature representation sequence of the text to be processed. Then, based on the feature representation sequence of the text to be processed, a summary of the text to be processed can be generated; for example, the feature representation sequence of the text to be processed is decoded to obtain a summary of the text to be processed.

[0042] In the data processing, a text task training dataset and a summary training dataset are acquired. The first model to be trained is trained using the text task training dataset to obtain N auxiliary task matrices. The side path of the first feedforward neural network layer of the first model to be trained has N training matrices. The N auxiliary task matrices are obtained by training the N training matrices using the text task training dataset. The second model to be trained is then called to perform summary generation processing on the sample text in the summary training dataset to obtain the summary generation result of the sample text. The N auxiliary task matrices are deployed in the side path of the gated fully connected layer of the second model to be trained, and the side path of the second feedforward neural network layer of the second model to be trained has an expert matrix. Based on the difference between the summary generation result and the target summary, the expert matrix is ​​adjusted to obtain the summary generation model. In the summary generation process, an intermediate sequence corresponding to the text to be processed is obtained. This intermediate sequence is obtained by encoding the associations between different text positions in the text. The results of at least two auxiliary tasks are weighted and fused to obtain a weighted result corresponding to the intermediate sequence. Different auxiliary task results are obtained by performing feature transformations on the intermediate sequence using different auxiliary task matrices. The weights corresponding to each auxiliary task result are determined based on the intermediate sequence. A nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed. Based on the feature representation sequence of the text to be processed, a summary of the text is generated. It is evident that by incorporating training auxiliary task matrices and decoupling the training processes of the auxiliary task matrices and expert matrices during data processing, the model to be trained can be targeted from multiple dimensions, thereby improving the accuracy of the text summarization generated by the summary generation model. By weightedly fusing the results of multiple auxiliary tasks during the summary generation process, the model can gain a more comprehensive and profound understanding of the text to be processed during the summary generation process, thus improving the accuracy of the text summarization generated by the summary generation model.

[0043] Based on the above data processing scheme and abstract generation scheme, this application proposes a more detailed data processing method and abstract generation method. The data processing method and abstract generation method proposed in this application will be described in detail below with reference to the accompanying drawings.

[0044] Please see Figure 3 , Figure 3 This is a flowchart illustrating a data processing method provided in an embodiment of this application. This data processing method can be executed by a computer device, which can be... Figure 1 The terminal device 101 or server 102 shown. Figure 3 As shown, the data processing method may include the following steps S301-S304: S301. Obtain the text task training dataset and the summary training dataset.

[0045] The text task training dataset includes N auxiliary task training datasets, where N is an integer greater than 1. Different auxiliary task training datasets are used to train different text task processing capabilities; these text tasks are all related to summarization generation, such as keyword extraction, sentence order prediction, partial summarization generation, and document importance scoring. The summarization training dataset includes sample text and target summaries; the target summaries are the labeled data of the sample text.

[0046] In one implementation, a computer device acquires N training texts and their labeled data, and then standardizes the N training texts and their labeled data according to an instruction set template to obtain N auxiliary task training datasets. The instruction set template includes a task instruction field, an input field, and a validation field; the value of the task instruction field in the i-th auxiliary task training dataset is generated based on the relationship between the i-th training text and its labeled data, where i is a positive integer less than or equal to N.

[0047] The instruction set template can be represented as: {"instruction": "...", "input": "...", "output": "..."}. The task instruction field (instruction) indicates the text task to be processed; it can also indicate the format of the output data. The input field (input) contains the training text, and the output field (output) contains the labeled data corresponding to the training text. For example, suppose the training text is "XX computation utilizes the superposition characteristics of XX bits to achieve parallel computation, its core advantage lies in the exponential increase in computing power brought about by XX entanglement and XX tunneling effects, completely breaking through the bottleneck of classical computers," and the labeled data corresponding to this training text is "XX computation; XX bits; XX entanglement; XX tunneling; parallel computation." From the training text and the labeled data corresponding to the training text, it can be seen that the labeled data corresponding to the training text consists of the keywords of the training text. Therefore, the value of the task instruction field can be: "Extract keywords from the given text and return a list of keywords"; optionally, the value of the task instruction field can also include: "Keywords in the keyword list are separated by ;".

[0048] In another implementation, the computer device acquires M related sub-texts and concatenates them to obtain sample text, where M is an integer greater than 1. After obtaining the sample text, the computer device further acquires the annotation data of the sample text. Then, the computer device performs standardization processing on the sample text and annotation data according to an instruction set template to obtain a summary training dataset; wherein, the instruction set template includes a task instruction field, an input field, and a validation field; the value of the task instruction field in the summary training dataset is used to describe the summary generation conditions.

[0049] For example, the sample text is obtained by concatenating three documents. Depending on the actual requirements, the task instruction field could be: "Please generate a comprehensive summary (100-200 words) based on the following three technical documents, covering core innovations, technical paths, and cross-document relevance." Correspondingly, the input field could be: "Document 1 'MoE Architecture Optimization': Proposes a dynamic gating sparse routing algorithm, improving inference speed by 40%...\nDocument 2 'Long Text Modeling': Utilizes displacement attention to achieve 32K context modeling, reducing memory usage by 62%...\nDocument 3 'Summary Generation System': Integrates MoE and displacement attention techniques, achieving a 12.7% improvement in ROUGE-L on the GovReport dataset." The value of the validation field can be: "Cross-document technology fusion drives breakthroughs in long text summarization:\n1. Architectural innovation: Combining dynamic gated sparse routing (document 1) and displacement attention (document 2), achieving a dual optimization of +40% inference speed and -62% memory usage;\n2. Performance leap: Achieving a ROUGE-L of 58.3 (document 3) on the GovReport dataset, a 12.7% improvement over the baseline;\n3. Technological connection: The 32K contextual capabilities of document 2 provide the foundation for document-level modeling of document 3, while the expert routing mechanism of document 1 supports multi-granularity semantic integration. This fusion scheme provides a new paradigm for the deployment of trillion-parameter summarization models." S302. Train the first model to be trained using the text task training dataset to obtain N auxiliary task matrices.

[0050] The first model to be trained includes a first feedforward neural network layer. The bypass of the first feedforward neural network layer has N matrices to be trained. The i-th auxiliary task matrix is ​​obtained by training the i-th matrix to be trained using the i-th auxiliary task training dataset, where i is a positive integer less than or equal to N.

[0051] In one implementation, the training dataset for the i-th auxiliary task includes the i-th training text and its labeled data. A computer device acquires N auxiliary task processing results output from N training matrices; wherein the i-th auxiliary task processing result is obtained based on the feature representation sequence of the i-th training text, and the feature representation sequence of the i-th training text is obtained by performing a linear transformation on the i-th training text using the i-th training matrix. The feature representation sequence of the i-th training text can be represented as:

[0052] in, Let be the feature representation sequence of the i-th training text. These are the backbone network weights of the first model to be trained. Freeze parameters during training. Let i represent the i-th matrix to be trained. Represents a universal matrix. Indicates the dimension of the output vector. Let i be the i-th training text.

[0053] In one embodiment, the first feedforward neural network layer includes a first fully connected projection matrix. The computer device acquires the input sequence of the first feedforward neural network layer; wherein the input sequence is obtained by encoding the association between any two text positions in the i-th training text. In one implementation, the first model to be trained further includes an attention layer, with a general matrix deployed in the sidebands of the attention layer. On one hand, the computer device calls the attention layer to encode the association between any two text positions in the i-th training text, obtaining the encoding result of the i-th training text; on the other hand, the computer device performs a linear transformation on the i-th training text using the general matrix, obtaining a first transformation result. After obtaining the encoding result and the first transformation result, the computer device uses the first transformation result to perform nonlinear mapping enhancement processing on the encoding result, obtaining the input sequence of the first feedforward neural network layer; for example, the computer device sums the encoding result and the first transformation result to obtain the input sequence of the first feedforward neural network layer.

[0054] After obtaining the input sequence of the first feedforward neural network layer, the computer device performs a nonlinear transformation on the input sequence using a first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text. In one implementation, on the one hand, the computer device performs a nonlinear transformation on the input sequence using the first fully connected projection matrix to obtain a second transformation result; on the other hand, the computer device uses the i-th training matrix to perform a linear transformation on the input sequence to obtain a third transformation result. After obtaining the second and third transformation results, the computer device uses the third transformation result to perform nonlinear mapping enhancement processing on the second transformation result to obtain the feature representation sequence of the i-th training text; for example, the computer device sums the second and third transformation results to obtain the feature representation sequence of the i-th training text. Then, based on the feature representation sequence of the i-th training text, the computer device generates the i-th auxiliary task processing result; for example, the computer device can decode the feature representation sequence of the i-th training text to obtain the i-th auxiliary task processing result.

[0055] Further, after obtaining the N auxiliary task processing results from the N training matrices as described above, the computer device constructs a first loss function based on the prediction bias of the N training matrices. The computer device sums the prediction biases of the N training matrices to obtain the first loss function; where the prediction bias of the i-th training matrix is ​​obtained based on the difference between the i-th auxiliary task processing result and the labeled data of the i-th training text. The i-th auxiliary task processing result is related to the i-th training matrix and the general matrix. The first loss function can be specifically expressed as:

[0056] in, To the number of auxiliary tasks, Let be the size of the training dataset for the i-th auxiliary task. For the training text in the dataset for the i-th auxiliary task, These are the backbone network weights of the first model to be trained, and these backbone network weights are frozen during training. For the i-th matrix to be trained, It is a general matrix.

[0057] After constructing the first loss function, the computer device adjusts the N training matrices to minimize the first loss function, resulting in N auxiliary task matrices. In one embodiment, the computer device adjusts the N training matrices and a general matrix to minimize the first loss function, resulting in N auxiliary task matrices and an adjusted general matrix. It should be noted that the general matrix is ​​used to assist in adjusting the N training matrices; the adjusted general matrix is ​​not deployed to the second training model.

[0058] Figure 4 This is a schematic diagram of the architecture of a first model to be trained, provided as an embodiment of this application. Figure 4 As shown, the first model to be trained can consist of several Transformer neural network architectures. The N training matrices involved in this application are deployed in the (first) feedforward neural network layer of the Transformer neural network architecture, while the general matrix is ​​deployed in the attention network layer of the Transformer neural network architecture. The general matrix primarily focuses on learning low-level feature representations, while the training matrices focus on high-level feature modeling for specific auxiliary tasks. When the general matrix and the training matrices are deployed in the same network layer, if their update directions are highly correlated, redundant parameters may be introduced; conversely, if their update directions are opposite, the existing knowledge representation of the original weights W may be corrupted. By decoupling the general matrix and the training matrix and deploying them in different network layers, conflicts in gradient directions caused by inconsistencies in their optimization objectives can be avoided, achieving efficient training.

[0059] In another implementation, the first feedforward neural network layer includes a first fully connected projection matrix; the i-th auxiliary task training dataset includes the i-th training text and its labeled data. A computer device acquires the input sequence of the first feedforward neural network layer and performs a nonlinear transformation on the input sequence using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text. Then, based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, the computer device adjusts the i-th training matrix to obtain the i-th auxiliary task matrix; wherein the auxiliary task processing result of the i-th training text is generated based on the feature representation sequence of the i-th training text.

[0060] In one embodiment, the first model to be trained further includes an attention layer, with a general matrix deployed in the side path of the attention layer. The computer device adjusts the general matrix and the i-th model to be trained based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, resulting in an adjusted general matrix and an i-th auxiliary task matrix.

[0061] S303. Call the second model to be trained to perform summary generation processing on the sample text to obtain the summary generation result of the sample text.

[0062] The second model to be trained includes a gated fully connected layer and a second feedforward neural network layer. N auxiliary task matrices are deployed in the bypass of the gated fully connected layer of the second model to be trained. In addition, an expert matrix is ​​deployed in the bypass of the second feedforward neural network layer.

[0063] In one implementation, the second model to be trained further includes an attention layer. In one embodiment, the computer device encodes the association between any two different text positions in the sample text using the attention layer to obtain an intermediate sequence corresponding to the sample text. In another embodiment, the attention layer is a shifted sparse attention layer. The shifted sparse attention layer divides the sample text into Q sub-sequence blocks according to a preset length, where Q is an integer greater than 1. Then, based on the association between different text positions in each sub-sequence block, the Q sub-sequence blocks are encoded to obtain the encoding results of the Q sub-sequence blocks; wherein, the encoding result of any sub-sequence block is obtained by encoding the association between any two text positions in that sub-sequence block. After obtaining the encoding results of the Q sub-sequence blocks, the encoding results of the Q sub-sequence blocks are passed across blocks using a sliding window to obtain the information transmission result of the sample text. Then, the encoding results of the Q sub-sequence blocks and the information transmission result of the sample text are aggregated to obtain the intermediate sequence corresponding to the sample text. Compared to traditional attention layers, shifted sparse attention layers can overcome the bottleneck of long text input (such as 20K token-level context), providing technical support for chapter-level summary generation.

[0064] After obtaining the intermediate sequence corresponding to the sample text, the computer device performs weighted fusion of the processing results of N auxiliary tasks through a gated fully connected layer to obtain the weighted result corresponding to the intermediate sequence. The N auxiliary task processing results are obtained by performing feature transformations on the intermediate sequence using N auxiliary task matrices. In one embodiment, the gated fully connected layer includes a second fully connected projection matrix. On one hand, the computer device projects the intermediate sequence corresponding to the sample text onto the gated space through the second fully connected projection matrix to obtain the weights corresponding to the N auxiliary task matrices. On the other hand, the computer device calls the N auxiliary task matrices to perform feature transformations on the intermediate sequence corresponding to the sample text, respectively, to obtain N auxiliary task processing results. The i-th auxiliary task processing result is obtained by performing feature transformations on the intermediate sequence corresponding to the sample text using the i-th auxiliary task matrix. After obtaining the weights corresponding to the N auxiliary task matrices and the N auxiliary task processing results, the computer device performs weighted fusion of the N auxiliary task processing results using the weights corresponding to the N auxiliary task matrices to obtain the weighted result corresponding to the sample text.

[0065] Further, the computer device invokes a second feedforward neural network layer to perform a nonlinear transformation on the weighted result corresponding to the sample text, obtaining a feature representation sequence of the sample text. In one embodiment, the second feedforward neural network layer includes a third fully connected projection matrix. On one hand, the computer device performs a nonlinear transformation on the weighted result corresponding to the sample text using the third fully connected projection matrix, obtaining a fourth transformation result; on the other hand, the computer device performs a linear transformation on the weighted result corresponding to the sample text using an expert matrix, obtaining a fifth transformation result. After obtaining the fourth and fifth transformation results, the fifth transformation result is used to perform nonlinear mapping enhancement processing on the fourth transformation result, obtaining a feature representation sequence of the sample text. Then, based on the feature representation sequence of the sample text, the computer device generates a summary of the sample text. The feature representation sequence of the sample text can specifically be represented as:

[0066] in, The feature representation sequence of the sample text. The weights are the backbone network weights of the second model to be trained (excluding gated fully connected layers). Freeze parameters during training. Representing the expert matrix, Indicates the dimension of the output vector. This is sample text. The weighted result corresponding to the sample text. This can be further expressed as:

[0067] in, For parameters of the gated fully connected layer, The parameters of the i-th auxiliary task matrix are... This represents the processing result of the i-th auxiliary task.

[0068] S304. Based on the difference between the generated summary and the target summary, the expert matrix is ​​adjusted to obtain the summary generation model.

[0069] The summary generation model is used to generate a text summary corresponding to the text to be processed. In one implementation, the computer device constructs a second loss function based on the difference between the summary generation result and the target summary, which can be specifically expressed as:

[0070] in, Indicates the size of the summary training dataset. These are the weights of the backbone network of the second model to be trained. These weights (excluding the gated fully connected layers) are frozen during training. There are N auxiliary task matrices, which are also frozen during training. For expert matrix, These are the parameters for the gated fully connected layer.

[0071] In one embodiment, the computer device adjusts the expert matrix and the second fully connected projection matrix to minimize a second loss function, thereby obtaining a summary generation model. The summary generation model includes the adjusted expert matrix and the adjusted second fully connected projection matrix. In another embodiment, the computer device adjusts the expert matrix to minimize a second loss function, thereby obtaining a summary generation model, which includes the adjusted expert matrix.

[0072] Figure 5 This is a schematic diagram of the architecture of a second model to be trained, provided as an embodiment of this application. Figure 5 As shown, the second model to be trained can consist of several Transformer neural network architectures. An additional gated fully connected layer is added between the attention network layer and the feedforward neural network layer. The N auxiliary task matrices obtained in step S202 are deployed in the bypass of the gated fully connected layer; in addition, an expert matrix is ​​also deployed in the bypass of the feedforward neural network layer. The goal of this architecture is to determine the weights of the processing results of the N auxiliary tasks based on the contextual information of the self-attention mechanism (i.e., the output of the attention network layer). The output obtained based on the gated fully connected layer and the N auxiliary task matrices directly affects the generation of subsequent feature representation sequences.

[0073] In this embodiment, a text task training dataset and a summary training dataset are obtained. The first model to be trained is trained using the text task training dataset to obtain N auxiliary task matrices. The first feedforward neural network layer of the first model to be trained has N training matrices deployed in its sidepath. These N auxiliary task matrices are obtained by training the N training matrices using the text task training dataset. A second model to be trained is then called to perform summary generation processing on sample text in the summary training dataset to obtain the summary generation result of the sample text. The N auxiliary task matrices are deployed in the sidepath of the gated fully connected layer of the second model to be trained, and an expert matrix is ​​deployed in the sidepath of the second feedforward neural network layer of the second model to be trained. Based on the difference between the summary generation result and the target summary, the expert matrix is ​​adjusted to obtain the summary generation model. It can be seen that by adding training auxiliary task matrices and decoupling the training processes of the auxiliary task matrices and expert matrices during data processing, the model to be trained can be targeted from multiple dimensions, thereby improving the accuracy of the text summaries generated by the summary generation model.

[0074] Please see Figure 6 , Figure 6This is a flowchart illustrating a digest generation method provided in an embodiment of this application. This digest generation method can be executed by a computer device, which can be... Figure 1 The terminal device 101 or server 102 shown. Figure 6 As shown, the abstract generation method may include the following steps S601-S604: S601. Obtain the intermediate sequence corresponding to the text to be processed.

[0075] The intermediate sequence corresponding to the text to be processed is obtained by encoding the associations between different text positions in the text to be processed. In one embodiment, a computer device acquires the text to be processed and encodes the associations between different text positions in the text to be processed through a shifted sparse attention layer in a summarization model to obtain the intermediate sequence corresponding to the text to be processed. In one embodiment, the shifted sparse attention layer divides the text to be processed into N sub-sequence blocks according to a preset length, where N is an integer greater than 1. Then, based on the associations between different text positions in each sub-sequence block, the N sub-sequence blocks are encoded to obtain the encoding results of the N sub-sequence blocks; wherein, the encoding result of any sub-sequence block is obtained by encoding the association between any two text positions in that sub-sequence block. Further, the encoding results of the N sub-sequence blocks are passed across blocks through a sliding window to obtain the information transmission result of the text to be processed. Then, the encoding results of the N sub-sequence blocks and the information transmission result of the text to be processed are aggregated to obtain the intermediate sequence corresponding to the text to be processed.

[0076] In another implementation, a computer device acquires the text to be processed and encodes the associations between any different text positions in the text to be processed through the attention network layer in the summary generation model to obtain the intermediate sequence corresponding to the text to be processed.

[0077] S602. Perform weighted fusion on the processing results of at least two auxiliary tasks to obtain the weighted result corresponding to the intermediate sequence.

[0078] The results of different auxiliary tasks are obtained by performing feature transformation on the intermediate sequence using different auxiliary task matrices, and the weights corresponding to each auxiliary task result are determined based on the intermediate sequence corresponding to the text to be processed.

[0079] In one implementation, the intermediate sequence is obtained by encoding the associations between different text positions in the text to be processed through the attention layer in the summarization generation model. The summarization generation model also includes a gated fully connected layer, which comprises a first fully connected projection matrix and at least two auxiliary task matrices. These auxiliary task matrices are all bypass matrices of the first fully connected projection matrix (i.e., the auxiliary matrices are all deployed in the bypass of the gated fully connected layer). The architecture of the summarization generation model can be referenced. Figure 5 .

[0080] On one hand, the computer device projects the intermediate sequence onto the gated space using a first fully connected projection matrix, obtaining the weights corresponding to each auxiliary task matrix. On the other hand, the computer device invokes at least two auxiliary task matrices to perform feature transformations on the intermediate sequence, obtaining at least two auxiliary task processing results. After obtaining the weights corresponding to each auxiliary task matrix and at least two auxiliary task processing results, the computer device performs weighted fusion of the corresponding auxiliary task processing results according to the weights corresponding to each auxiliary task matrix, obtaining the weighted result corresponding to the intermediate sequence.

[0081] S603. Perform a nonlinear transformation on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed.

[0082] In one implementation, the summary generation model further includes a feedforward neural network layer, which comprises a second fully connected projection matrix and an expert matrix, wherein the expert matrix is ​​a bypass matrix of the second fully connected projection matrix (i.e., the expert matrix is ​​deployed as a bypass of the feedforward neural network layer). On one hand, the computer device performs a nonlinear transformation on the weighted result corresponding to the intermediate sequence using the second fully connected projection matrix to obtain a first transformation result; on the other hand, the computer device performs a linear transformation on the weighted result corresponding to the intermediate sequence using the expert matrix to obtain a second transformation result. After obtaining the first and second transformation results, the computer device uses the second transformation result to perform nonlinear mapping enhancement processing on the first transformation result to obtain a feature representation sequence of the text to be processed; for example, the computer device sums the first and second transformation results to obtain a feature representation sequence of the text to be processed.

[0083] S604. Generate a summary of the text to be processed based on the feature representation sequence of the text to be processed.

[0084] In one implementation, a computer device can decode a sequence of feature representations of the text to be processed to obtain a summary of the text.

[0085] In this embodiment, an intermediate sequence corresponding to the text to be processed is obtained. This intermediate sequence is obtained by encoding the associations between different text positions within the text. The results of at least two auxiliary tasks are weighted and fused to obtain a weighted result corresponding to the intermediate sequence. Different auxiliary task results are obtained by performing feature transformations on the intermediate sequence using different auxiliary task matrices. The weights corresponding to each auxiliary task result are determined based on the intermediate sequence. A nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain a feature representation sequence of the text to be processed. Based on this feature representation sequence, a summary of the text to be processed is generated. Therefore, by weightedly fusing the results of multiple auxiliary tasks, the model can gain a more comprehensive and profound understanding of the text to be processed during the summary generation process, thereby improving the accuracy of the text summary generated by the summary generation model.

[0086] Please see Figure 7 , Figure 7 This is a flowchart illustrating another abstract generation method provided in an embodiment of this application. This abstract generation method can be executed by a computer device, which can be... Figure 1 The terminal device 101 or server 102 shown. Figure 7 As shown, the abstract generation method may include the following steps S701-S706: S701. Obtain the text task training dataset and the summary training dataset.

[0087] S702. Train the first model to be trained using the text task training dataset to obtain N auxiliary task matrices.

[0088] S703. Call the second model to be trained to perform summary generation processing on the sample text to obtain the summary generation result of the sample text.

[0089] S704. Based on the difference between the generated summary and the target summary, the expert matrix is ​​adjusted to obtain the summary generation model.

[0090] For detailed implementation methods of steps S701-S704, please refer to [link / reference]. Figure 3 The implementation methods for steps S301-S304 will not be described in detail here.

[0091] S705. Obtain the text to be processed.

[0092] S706. Call the summary generation model to generate a summary of the text to be processed, and obtain a summary of the text to be processed.

[0093] For detailed implementation methods of steps S705 and S706, please refer to... Figure 6 The implementation methods for steps S601-S604 are not described in detail here.

[0094] Figure 8 This is a schematic diagram illustrating an embodiment of the present application for abstract generation. Figure 8 As shown, the computer device first constructs a text task training dataset and a summary training dataset according to an instruction set template. Then, it trains N training matrices and a general matrix using the text task training dataset to obtain N auxiliary task matrices. Next, the N auxiliary task matrices are deployed into the training model, which also contains the general matrix. Further, the training model is trained using the summary training dataset to obtain a summary generation model. This summary generation model can then generate a summary of the text to be processed.

[0095] To verify the effectiveness of the summary generation method provided in this application, the applicant trained and evaluated the method on the GovReport document-level summarization dataset, sampling 20,000 samples for training and 2,200 test samples for evaluation, with text lengths ranging from 20k to 64k tokens. In practical applications, the model to be trained can be the Llama2-7B model. Table 1 shows a comparison of the effectiveness provided by the embodiments of this application.

[0096] Table 1

[0097] As shown in Table 1, the summarization method provided in this application achieves a ROUGE-L of 58.3 for 64K text, a 17.1% improvement over the baseline; simultaneously, the performance degradation during the 20K to 64K length expansion is controlled within 2.1%, demonstrating length robustness; and the training cost is reduced by 64% (GPU days from 14.3 days to 5.2 days). This demonstrates the effectiveness of the summarization method provided in this application for long document summarization tasks. Table 2 is a comparison chart of the summarization method provided in this application's embodiments and alternative solutions.

[0098] Table 2

[0099] Option 1 involves removing the gated fully connected layer from the summary generation model, with the input data for the feedforward neural network layer being the result of a randomly selected auxiliary task. Option 2 involves deploying the N auxiliary task matrices and the expert matrix in the bypass of the same network layer. Option 3 involves using a multi-task learning method (i.e., not decoupling the training process into two stages: auxiliary task matrix training and expert matrix training). As shown in Table 2, the gated fully connected layer contributes approximately 5.3% to cross-task knowledge integration; hierarchical expert deployment effectively reduces task conflict interference, further improving ROUGE-L by 8.9%; and the continuous learning strategy (i.e., staged training) significantly improves learning performance compared to the multi-task learning approach that mixes all tasks together.

[0100] In the data processing, a text task training dataset and a summary training dataset are acquired. The first model to be trained is trained using the text task training dataset to obtain N auxiliary task matrices. The side path of the first feedforward neural network layer of the first model to be trained has N training matrices. The N auxiliary task matrices are obtained by training the N training matrices using the text task training dataset. The second model to be trained is then called to perform summary generation processing on the sample text in the summary training dataset to obtain the summary generation result of the sample text. The N auxiliary task matrices are deployed in the side path of the gated fully connected layer of the second model to be trained, and the side path of the second feedforward neural network layer of the second model to be trained has an expert matrix. Based on the difference between the summary generation result and the target summary, the expert matrix is ​​adjusted to obtain the summary generation model. In the summary generation process, an intermediate sequence corresponding to the text to be processed is obtained. This intermediate sequence is obtained by encoding the associations between different text positions in the text. The results of at least two auxiliary tasks are weighted and fused to obtain a weighted result corresponding to the intermediate sequence. Different auxiliary task results are obtained by performing feature transformations on the intermediate sequence using different auxiliary task matrices. The weights corresponding to each auxiliary task result are determined based on the intermediate sequence. A nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed. Based on the feature representation sequence of the text to be processed, a summary of the text is generated. It is evident that by incorporating training auxiliary task matrices and decoupling the training processes of the auxiliary task matrices and expert matrices during data processing, the model to be trained can be targeted from multiple dimensions, thereby improving the accuracy of the text summarization generated by the summary generation model. By weightedly fusing the results of multiple auxiliary tasks during the summary generation process, the model can gain a more comprehensive and profound understanding of the text to be processed during the summary generation process, thus improving the accuracy of the text summarization generated by the summary generation model.

[0101] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0102] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Figure 9 The data processing device shown can be mounted in a computer device, which may specifically be a server or a terminal device. Figure 9 The data processing apparatus shown can be used to perform the above. Figure 3 and Figure 7 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 9 The data processing device includes: The acquisition unit 901 is used to acquire the text task training dataset and the summary training dataset. The text task training dataset includes N auxiliary task training datasets, and the summary training dataset includes sample text and target summary. The target summary is the labeled data of the sample text, and N is an integer greater than 1. Processing unit 902 is used to train the first model to be trained using a text task training dataset to obtain N auxiliary task matrices. The first model to be trained includes a first feedforward neural network layer. The first feedforward neural network layer has N training matrices deployed in the bypass. The i-th auxiliary task matrix is ​​obtained by training the i-th training matrix using the i-th auxiliary task training dataset. i is a positive integer less than or equal to N. And it is used to call the second model to be trained to perform summary generation processing on the sample text and obtain the summary generation result of the sample text; the second model to be trained includes a gated fully connected layer and a second feedforward neural network layer, N auxiliary task matrices are deployed in the bypass of the gated fully connected layer, and an expert matrix is ​​deployed in the bypass of the second feedforward neural network layer; And it is used to adjust the expert matrix based on the difference between the summary generation result and the target summary to obtain the summary generation model, which is used to generate text summaries.

[0103] In one implementation, the i-th auxiliary task training dataset includes the i-th training text and the labeled data of the i-th training text; the processing unit 902 is used to train the first model to be trained using the text task training dataset to obtain N auxiliary task matrices, specifically for: Obtain N auxiliary task processing results from the output of N training matrices. The i-th auxiliary task processing result is obtained based on the feature representation sequence of the i-th training text. The feature representation sequence of the i-th training text is obtained by performing a linear transformation on the i-th training text based on the i-th training matrix. The prediction biases of the N training matrices are summed to obtain the first loss function; the prediction bias of the i-th training matrix is ​​obtained based on the difference between the processing result of the i-th auxiliary task and the labeled data of the i-th training text. With the goal of minimizing the first loss function, the N training matrices are adjusted to obtain N auxiliary task matrices.

[0104] In one embodiment, the first feedforward neural network layer includes a first fully connected projection matrix; the processing unit 902 is used to obtain N auxiliary task processing results output by N matrices to be trained, specifically for: Obtain the input sequence of the first feedforward neural network layer. The input sequence is obtained by encoding the association between any two text positions in the i-th training text. The input sequence is nonlinearly transformed using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text. Based on the feature representation sequence of the i-th training text, generate the i-th auxiliary task processing result.

[0105] In one implementation, the first model to be trained further includes an attention layer, the sidepath of which is configured with a general matrix; the processing unit 902 is used to acquire the input sequence of the first feedforward neural network layer, specifically for: The attention layer is invoked to encode the association between any two text positions in the i-th training text, thus obtaining the encoding result of the i-th training text; The first transformation result is obtained by performing a linear transformation on the i-th training text using a general matrix. The encoding result is enhanced by nonlinear mapping using the first transformation result to obtain the input sequence of the first feedforward neural network layer.

[0106] In one embodiment, the processing unit 902 is configured to perform a nonlinear transformation on the input sequence using a first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text, specifically configured to: The input sequence is nonlinearly transformed using the first fully connected projection matrix to obtain the second transformation result; The i-th matrix to be trained is used to perform a linear transformation on the input sequence to obtain the third transformation result; The second transformation result is enhanced by nonlinear mapping using the third transformation result to obtain the feature representation sequence of the i-th training text.

[0107] In one implementation, the first model to be trained further includes an attention layer, the sidepath of which is configured with a general matrix; the processing unit 902 is used to adjust the N training matrices to obtain N auxiliary task matrices with the objective of minimizing a first loss function, specifically for: With the goal of minimizing the first loss function, the general matrix and N training matrices are adjusted to obtain the adjusted general matrix and N auxiliary task matrices.

[0108] In one implementation, the first feedforward neural network layer includes a first fully connected projection matrix; the i-th auxiliary task training dataset includes the i-th training text and the labeled data of the i-th training text; the processing unit 902 is used to train the first model to be trained using the text task training dataset to obtain N auxiliary task matrices, specifically for: Obtain the input sequence of the first feedforward neural network layer. The input sequence is obtained by encoding the association between any two text positions in the i-th training text. The input sequence is nonlinearly transformed using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text. Based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, the i-th training matrix is ​​adjusted to obtain the i-th auxiliary task matrix; the auxiliary task processing result of the i-th training text is generated based on the feature representation sequence of the i-th training text.

[0109] In one implementation, the first model to be trained further includes an attention layer, the side path of which is a general matrix; the processing unit 902 is used to adjust the i-th matrix to be trained based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, to obtain the i-th auxiliary task matrix, specifically for: Based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, the general matrix and the i-th training matrix are adjusted to obtain the adjusted general matrix and the i-th auxiliary task matrix.

[0110] In one implementation, the second model to be trained further includes an attention layer; the processing unit 902 is used to call the second model to be trained to perform summary generation processing on the sample text, and obtain the summary generation result of the sample text, specifically for: By encoding the associations between different text positions in the sample text through an attention layer, the intermediate sequence corresponding to the sample text is obtained; The weighted result of the sample text is obtained by weighted fusion of the processing results of N auxiliary tasks through a gated fully connected layer; the processing results of N auxiliary tasks are obtained by feature transformation of the intermediate sequence using N auxiliary task matrices. The second feedforward neural network layer is invoked to perform a nonlinear transformation on the weighted result corresponding to the sample text, thereby obtaining the feature representation sequence of the sample text. Based on the feature representation sequence of the sample text, a summary of the sample text is generated.

[0111] In one implementation, the gated fully connected layer includes a second fully connected projection matrix; the processing unit 902 is used to adjust the expert matrix based on the difference between the summary generation result and the target summary to obtain a summary generation model, specifically for: A second loss function is constructed based on the difference between the generated summary and the target summary. With the goal of minimizing the second loss function, the expert matrix and the second fully connected projection matrix are adjusted to obtain the summary generation model, which includes the adjusted expert matrix and the adjusted second fully connected projection matrix.

[0112] In one embodiment, the processing unit 902 is configured to acquire a text task training dataset, specifically for: Obtain N training texts and their labeled data; The N training texts and their labeled data are standardized according to the instruction set template to obtain N auxiliary task training datasets. The instruction set template includes a task instruction field, an input field, and a verification field; the value of the task instruction field of the i-th auxiliary task training dataset is generated based on the relationship between the i-th training text and the labeled data of the i-th training text.

[0113] In one implementation, the processing unit 902 is configured to acquire a summary training dataset, specifically for: Get M related sub-texts and concatenate them to obtain the sample text, where M is an integer greater than 1; Obtain the labeled data of the sample text; The sample text and labeled data are standardized according to the instruction set template to obtain the summary training dataset. The instruction set template includes a task instruction field, an input field, and a validation field; the value of the task instruction field in the summary training dataset is used to describe the summary generation conditions.

[0114] According to one embodiment of this application, Figure 3 and Figure 7 The data processing method shown may involve some steps that can be derived from... Figure 9 The data processing apparatus shown is executed by each unit within it. For example, Figure 3 Step S301 shown can be performed by Figure 9 The acquisition unit 901 shown is executed, and steps S302-S304 can be performed by... Figure 9 The processing unit 902 shown executes the operation; Figure 7 Steps S701 and S705 shown can be derived from... Figure 9 The acquisition unit 901 shown is executed, and steps S702-S704 and S706 can be performed by... Figure 9 The processing unit 902 shown executes. Figure 9The data processing apparatus shown can be constructed by combining each unit individually or entirely into one or more other units, or one or more of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0115] According to another embodiment of this application, a general-purpose computing device, such as a computer device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can perform operations such as... Figure 3 and Figure 7 The computer program (including program code) involved in each step of the corresponding method shown, to construct such... Figure 9 The data processing apparatus shown herein, and the data processing method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and executed therein.

[0116] Based on the same inventive concept, the principle and beneficial effects of the data processing device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the data processing method in the embodiments of this application in solving the problem. For the sake of brevity, the principle and beneficial effects of the method implementation can be referred to.

[0117] Please see Figure 10 , Figure 10 This is a schematic diagram of a summary generation device provided in an embodiment of this application. Figure 10 The summary generation device shown can be mounted in a computer device, which can specifically be a server or a terminal device. Figure 10 The summary generation apparatus shown can be used to perform the above. Figure 6 and Figure 7 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 10 The abstract generation apparatus includes: The acquisition unit 1001 is used to acquire the intermediate sequence corresponding to the text to be processed. The intermediate sequence is obtained by encoding the association between different text positions in the text to be processed. The processing unit 1002 is used to perform weighted fusion of the processing results of at least two auxiliary tasks to obtain the weighted result corresponding to the intermediate sequence; the processing results of different auxiliary tasks are obtained by performing feature transformation on the intermediate sequence using different auxiliary task matrices, and the weights corresponding to each processing result of auxiliary tasks are determined based on the intermediate sequence. And a nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed; And a summary of the text to be processed, based on the feature representation sequence of the text to be processed.

[0118] In one implementation, the intermediate sequence is obtained by encoding the association between different text positions in the text to be processed through the attention layer in the summary generation model; the summary generation model also includes a gated fully connected layer, which includes a first fully connected projection matrix and at least two auxiliary task matrices, wherein the at least two auxiliary task matrices are bypass matrices of the first fully connected projection matrix. Processing unit 1002 is used to perform weighted fusion of the processing results of at least two auxiliary tasks to obtain a weighted result corresponding to the intermediate sequence, specifically for: The intermediate sequence is projected onto the gate space using the first fully connected projection matrix to obtain the weights corresponding to each auxiliary task matrix; Call at least two auxiliary task matrices to perform feature transformation on the intermediate sequence respectively, and obtain at least two auxiliary task processing results; The processing results of the corresponding auxiliary tasks are weighted and fused according to the weights of each auxiliary task matrix to obtain the weighted result corresponding to the intermediate sequence.

[0119] In one implementation, the intermediate sequence is obtained by encoding the association between different text positions in the text to be processed through the attention layer in the summarization generation model; the summarization generation model further includes a feedforward neural network layer, which includes a second fully connected projection matrix and an expert matrix, wherein the expert matrix is ​​a bypass matrix of the second fully connected projection matrix; the processing unit 1002 is used to perform a nonlinear transformation on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed, specifically for: The first transformation result is obtained by performing a nonlinear transformation on the weighted result corresponding to the intermediate sequence using the second fully connected projection matrix. The second transformation result is obtained by linearly transforming the weighted result corresponding to the intermediate sequence using the expert matrix; The first transformation result is enhanced by nonlinear mapping using the second transformation result to obtain the feature representation sequence of the text to be processed.

[0120] In one embodiment, the processing unit 1002 is configured to obtain an intermediate sequence corresponding to the text to be processed, specifically for: Obtain the text to be processed and divide it into N sub-sequence blocks according to a preset length, where N is an integer greater than 1; Based on the association between different text positions in each sub-sequence block, N sub-sequence blocks are encoded to obtain the encoding results of N sub-sequence blocks; the encoding result of any sub-sequence block is obtained by encoding the association between any two text positions in that sub-sequence block. The encoding results of N sub-sequence blocks are passed across blocks using a sliding window to obtain the information transmission result of the text to be processed. The encoding results of N sub-sequence blocks and the information transmission results of the text to be processed are aggregated to obtain the intermediate sequence corresponding to the text to be processed.

[0121] According to one embodiment of this application, Figure 6 and Figure 7 The abstract generation method shown can be partially implemented by... Figure 10 The summarization generation apparatus shown is executed by individual units within it. For example, Figure 6 Step S601 shown can be performed by Figure 10 The acquisition unit 1001 shown is executed, and steps S602-S604 can be performed by... Figure 10 The processing unit 1002 shown executes this; Figure 7 Steps S701 and S705 shown can be derived from... Figure 10 The acquisition unit 1001 shown is executed, and steps S702-S704 and S706 can be performed by... Figure 10 The processing unit 1002 shown is executed. Figure 10 The units in the illustrated summary generation apparatus can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the summary generation apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0122] According to another embodiment of this application, a general-purpose computing device, such as a computer device including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can perform operations such as... Figure 6 and Figure 7The computer program (including program code) involved in each step of the corresponding method shown, to construct such... Figure 10 The abstract generation apparatus shown herein, and the abstract generation method for implementing the embodiments of this application, are described. A computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and executed therein.

[0123] Based on the same inventive concept, the principle and beneficial effects of the abstract generation device provided in the embodiments of this application are similar to the principle and beneficial effects of the abstract generation method in the embodiments of this application. For details, please refer to the principle and beneficial effects of the method implementation. For the sake of brevity, these will not be repeated here.

[0124] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. The computer device may be a terminal device or a server. Figure 11 As shown, the computer device includes at least a processor 1101, a communication interface 1102, and a memory 1103. The processor 1101, communication interface 1102, and memory 1103 can be connected via a bus or other means. The processor 1101 (or Central Processing Unit, CPU) is the computing and control core of the computer device. It can parse various instructions within the computer device and process various data. For example, the CPU can parse power-on / off commands issued by objects to the computer device and control the computer device to perform power-on / off operations; it can also transmit various interactive data between internal structures of the computer device, and so on. The communication interface 1102 may optionally include a standard wired interface or a wireless interface (such as Wi-Fi, mobile communication interface, etc.), and can be used to send and receive data under the control of the processor 1101; the communication interface 1102 can also be used for data transmission and interaction within the computer device. The memory 1103 is the storage device in the computer device, used to store programs and data. It is understood that the memory 1103 here may include the built-in memory of the computer device, or it may include the extended memory supported by the computer device. The memory 1103 provides storage space for storing the operating system of the computer device, which may include, but is not limited to, Android, iOS, Windows Phone, etc., and this application does not limit this.

[0125] This application embodiment also provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device. Furthermore, the storage space also stores computer programs suitable for loading and execution by the processor 1101. It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0126] In one embodiment, the processor 1101 performs the following operations by running a computer program stored in memory 1103: Obtain the text task training dataset and the summary training dataset. The text task training dataset includes N auxiliary task training datasets. The summary training dataset includes sample text and target summary. The target summary is the labeled data of the sample text. N is an integer greater than 1. The first model to be trained is trained using a text task training dataset to obtain N auxiliary task matrices. The first model to be trained includes a first feedforward neural network layer, and the sidebar of the first feedforward neural network layer has N training matrices. The i-th auxiliary task matrix is ​​obtained by training the i-th auxiliary task matrix using the i-th auxiliary task training dataset, where i is a positive integer less than or equal to N. The second model to be trained is called to perform a summary generation process on the sample text, and the summary generation result of the sample text is obtained. The second model to be trained includes a gated fully connected layer and a second feedforward neural network layer. N auxiliary task matrices are deployed in the bypass of the gated fully connected layer, and expert matrices are deployed in the bypass of the second feedforward neural network layer. Based on the difference between the generated summary and the target summary, the expert matrix is ​​adjusted to obtain the summary generation model, which is used to generate text summaries.

[0127] As an optional embodiment, the i-th auxiliary task training dataset includes the i-th training text and the labeled data of the i-th training text; the processor 1101 trains the first model to be trained using the text task training dataset to obtain N auxiliary task matrices. A specific embodiment is as follows: Obtain N auxiliary task processing results from the output of N training matrices. The i-th auxiliary task processing result is obtained based on the feature representation sequence of the i-th training text. The feature representation sequence of the i-th training text is obtained by performing a linear transformation on the i-th training text based on the i-th training matrix. The prediction biases of the N training matrices are summed to obtain the first loss function; the prediction bias of the i-th training matrix is ​​obtained based on the difference between the processing result of the i-th auxiliary task and the labeled data of the i-th training text. With the goal of minimizing the first loss function, the N training matrices are adjusted to obtain N auxiliary task matrices.

[0128] As an optional embodiment, the first feedforward neural network layer includes a first fully connected projection matrix; a specific embodiment in which the processor 1101 obtains the N auxiliary task processing results output by the N matrices to be trained is as follows: Obtain the input sequence of the first feedforward neural network layer. The input sequence is obtained by encoding the association between any two text positions in the i-th training text. The input sequence is nonlinearly transformed using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text. Based on the feature representation sequence of the i-th training text, generate the i-th auxiliary task processing result.

[0129] As an optional embodiment, the first model to be trained further includes an attention layer, the side path of which is equipped with a general matrix; a specific embodiment of the processor 1101 acquiring the input sequence of the first feedforward neural network layer is as follows: The attention layer is invoked to encode the association between any two text positions in the i-th training text, thus obtaining the encoding result of the i-th training text; The first transformation result is obtained by performing a linear transformation on the i-th training text using a general matrix. The encoding result is enhanced by nonlinear mapping using the first transformation result to obtain the input sequence of the first feedforward neural network layer.

[0130] As an optional embodiment, the processor 1101 performs a nonlinear transformation on the input sequence using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text. A specific embodiment of this is as follows: The input sequence is nonlinearly transformed using the first fully connected projection matrix to obtain the second transformation result; The i-th matrix to be trained is used to perform a linear transformation on the input sequence to obtain the third transformation result; The second transformation result is enhanced by nonlinear mapping using the third transformation result to obtain the feature representation sequence of the i-th training text.

[0131] As an optional embodiment, the first model to be trained further includes an attention layer, the side of which is equipped with a general matrix; the processor 1101 adjusts the N training matrices to obtain N auxiliary task matrices with the objective of minimizing the first loss function. A specific embodiment is as follows: With the goal of minimizing the first loss function, the general matrix and N training matrices are adjusted to obtain the adjusted general matrix and N auxiliary task matrices.

[0132] As an optional embodiment, the first feedforward neural network layer includes a first fully connected projection matrix; the i-th auxiliary task training dataset includes the i-th training text and the labeled data of the i-th training text; the processor 1101 trains the first model to be trained using the text task training dataset to obtain N auxiliary task matrices. A specific embodiment is as follows: Obtain the input sequence of the first feedforward neural network layer. The input sequence is obtained by encoding the association between any two text positions in the i-th training text. The input sequence is nonlinearly transformed using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text. Based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, the i-th training matrix is ​​adjusted to obtain the i-th auxiliary task matrix; the auxiliary task processing result of the i-th training text is generated based on the feature representation sequence of the i-th training text.

[0133] As an optional embodiment, the first model to be trained further includes an attention layer, with a general matrix deployed in the side path of the attention layer; the processor 1101 adjusts the i-th matrix to be trained based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, and the specific embodiment of obtaining the i-th auxiliary task matrix is ​​as follows: Based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, the general matrix and the i-th training matrix are adjusted to obtain the adjusted general matrix and the i-th auxiliary task matrix.

[0134] As an optional embodiment, the second model to be trained further includes an attention layer; the processor 1101 calls the second model to be trained to perform summary generation processing on the sample text to obtain the summary generation result of the sample text. A specific embodiment is as follows: By encoding the associations between different text positions in the sample text through an attention layer, the intermediate sequence corresponding to the sample text is obtained; The weighted result of the sample text is obtained by weighted fusion of the processing results of N auxiliary tasks through a gated fully connected layer; the processing results of N auxiliary tasks are obtained by feature transformation of the intermediate sequence using N auxiliary task matrices. The second feedforward neural network layer is invoked to perform a nonlinear transformation on the weighted result corresponding to the sample text, thereby obtaining the feature representation sequence of the sample text. Based on the feature representation sequence of the sample text, a summary of the sample text is generated.

[0135] As an optional embodiment, the gated fully connected layer includes a second fully connected projection matrix; the processor 1101 adjusts the expert matrix based on the difference between the summary generation result and the target summary, resulting in the following specific embodiment of the summary generation model: A second loss function is constructed based on the difference between the generated summary and the target summary. With the goal of minimizing the second loss function, the expert matrix and the second fully connected projection matrix are adjusted to obtain the summary generation model, which includes the adjusted expert matrix and the adjusted second fully connected projection matrix.

[0136] As an optional embodiment, the processor 1101 acquires the text task training dataset in the following specific embodiment: Obtain N training texts and their labeled data; The N training texts and their labeled data are standardized according to the instruction set template to obtain N auxiliary task training datasets. The instruction set template includes a task instruction field, an input field, and a verification field; the value of the task instruction field of the i-th auxiliary task training dataset is generated based on the relationship between the i-th training text and the labeled data of the i-th training text.

[0137] As an optional embodiment, the processor 1101 acquires the summary training dataset in the following specific embodiment: Get M related sub-texts and concatenate them to obtain the sample text, where M is an integer greater than 1; Obtain the labeled data of the sample text; The sample text and labeled data are standardized according to the instruction set template to obtain the summary training dataset. The instruction set template includes a task instruction field, an input field, and a validation field; the value of the task instruction field in the summary training dataset is used to describe the summary generation conditions.

[0138] In another embodiment, processor 1101 performs the following operations by running a computer program in memory 1103: Obtain the intermediate sequence corresponding to the text to be processed. The intermediate sequence is obtained by encoding the relationship between different text positions in the text to be processed. The results of at least two auxiliary tasks are weighted and fused to obtain the weighted result corresponding to the intermediate sequence. The results of different auxiliary tasks are obtained by performing feature transformation on the intermediate sequence using different auxiliary task matrices. The weights corresponding to each auxiliary task result are determined based on the intermediate sequence. A nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed. A summary of the text to be processed is generated based on the feature representation sequence of the text to be processed.

[0139] As an optional embodiment, the intermediate sequence is obtained by encoding the association between different text positions in the text to be processed through the attention layer in the summary generation model; the summary generation model also includes a gated fully connected layer, which includes a first fully connected projection matrix and at least two auxiliary task matrices, wherein the at least two auxiliary task matrices are both bypass matrices of the first fully connected projection matrix. A specific embodiment of processor 1101 performing weighted fusion of at least two auxiliary task processing results to obtain the weighted result corresponding to the intermediate sequence is as follows: The intermediate sequence is projected onto the gate space using the first fully connected projection matrix to obtain the weights corresponding to each auxiliary task matrix; Call at least two auxiliary task matrices to perform feature transformation on the intermediate sequence respectively, and obtain at least two auxiliary task processing results; The processing results of the corresponding auxiliary tasks are weighted and fused according to the weights of each auxiliary task matrix to obtain the weighted result corresponding to the intermediate sequence.

[0140] As an optional embodiment, the intermediate sequence is obtained by encoding the association between different text positions in the text to be processed through the attention layer in the summarization generation model; the summarization generation model also includes a feedforward neural network layer, which includes a second fully connected projection matrix and an expert matrix, wherein the expert matrix is ​​a bypass matrix of the second fully connected projection matrix; The processor 1101 performs a nonlinear transformation on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed. A specific embodiment is as follows: The first transformation result is obtained by performing a nonlinear transformation on the weighted result corresponding to the intermediate sequence using the second fully connected projection matrix. The second transformation result is obtained by linearly transforming the weighted result corresponding to the intermediate sequence using the expert matrix; The first transformation result is enhanced by nonlinear mapping using the second transformation result to obtain the feature representation sequence of the text to be processed.

[0141] As an optional embodiment, the processor 1101 obtains the intermediate sequence corresponding to the text to be processed in the following specific embodiment: Obtain the text to be processed and divide it into N sub-sequence blocks according to a preset length, where N is an integer greater than 1; Based on the association between different text positions in each sub-sequence block, N sub-sequence blocks are encoded to obtain the encoding results of N sub-sequence blocks; the encoding result of any sub-sequence block is obtained by encoding the association between any two text positions in that sub-sequence block. The encoding results of N sub-sequence blocks are passed across blocks using a sliding window to obtain the information transmission result of the text to be processed. The encoding results of N sub-sequence blocks and the information transmission results of the text to be processed are aggregated to obtain the intermediate sequence corresponding to the text to be processed.

[0142] Based on the same inventive concept, the principle and beneficial effects of the computer device provided in the embodiments of this application in solving the problem are similar to the principle and beneficial effects of the data processing method or summary generation method in the embodiments of this application in solving the problem. Please refer to the principle and beneficial effects of the implementation of the method. For the sake of brevity, they will not be repeated here.

[0143] This application also provides a computer-readable storage medium storing a computer program, which is adapted to be loaded by a processor and execute the data processing method of the above method embodiments; or, adapted to be loaded by a processor and execute the summary generation method of the above method embodiments.

[0144] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data processing method described above; or, causing the computer device to perform the digest generation method described above.

[0145] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0146] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0147] In the embodiments of this application, the term "module" or "unit" refers to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0148] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0149] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art will understand that all or part of the processes for implementing the above embodiments and equivalent variations made in accordance with the claims of this application are still within the scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a text task training dataset and a summary training dataset. The text task training dataset includes N auxiliary task training datasets. The summary training dataset includes sample text and target summary. The target summary is the labeled data of the sample text. N is an integer greater than 1. The first model to be trained is trained using the text task training dataset to obtain N auxiliary task matrices. The first model to be trained includes a first feedforward neural network layer. The first feedforward neural network layer has N training matrices deployed in the bypass. The i-th auxiliary task matrix is ​​obtained by training the i-th training matrix using the i-th auxiliary task training dataset. i is a positive integer less than or equal to N. The second model to be trained is invoked to perform a summary generation process on the sample text, and the summary generation result of the sample text is obtained; the second model to be trained includes a gated fully connected layer and a second feedforward neural network layer, the N auxiliary task matrices are deployed in the bypass of the gated fully connected layer, and the bypass of the second feedforward neural network layer is deployed with an expert matrix; Based on the difference between the generated summary and the target summary, the expert matrix is ​​adjusted to obtain a summary generation model, which is used to generate text summaries.

2. The method as described in claim 1, characterized in that, The i-th auxiliary task training dataset includes the i-th training text and the labeled data of the i-th training text; The first model to be trained is trained using the text task training dataset to obtain N auxiliary task matrices, including: Obtain N auxiliary task processing results output by the N training matrices. The i-th auxiliary task processing result is obtained based on the feature representation sequence of the i-th training text. The feature representation sequence of the i-th training text is obtained by performing a linear transformation on the i-th training text based on the i-th training matrix. The prediction biases of the N training matrices are summed to obtain the first loss function; the prediction bias of the i-th training matrix is ​​obtained based on the difference between the processing result of the i-th auxiliary task and the labeled data of the i-th training text. With the goal of minimizing the first loss function, the N training matrices are adjusted to obtain N auxiliary task matrices.

3. The method as described in claim 2, characterized in that, The first feedforward neural network layer includes a first fully connected projection matrix; obtaining the N auxiliary task processing results output by the N training matrices includes: Obtain the input sequence of the first feedforward neural network layer, wherein the input sequence is obtained by encoding the association between any two text positions in the i-th training text; The input sequence is nonlinearly transformed using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text; Based on the feature representation sequence of the i-th training text, the i-th auxiliary task processing result is generated.

4. The method as described in claim 3, characterized in that, The first model to be trained further includes an attention layer, wherein a general matrix is ​​deployed in the bypass of the attention layer; obtaining the input sequence of the first feedforward neural network layer includes: The attention layer is invoked to encode the association between any two text positions in the i-th training text, thereby obtaining the encoding result of the i-th training text; A first transformation result is obtained by performing a linear transformation on the i-th training text using the general matrix. The first transformation result is used to perform nonlinear mapping enhancement processing on the encoding result to obtain the input sequence of the first feedforward neural network layer.

5. The method as described in claim 3, characterized in that, The step of performing a nonlinear transformation on the input sequence using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text includes: The input sequence is nonlinearly transformed using the first fully connected projection matrix to obtain the second transformation result; The input sequence is linearly transformed by the i-th matrix to be trained, resulting in a third transformation. The second transformation result is then subjected to nonlinear mapping enhancement processing using the third transformation result to obtain the feature representation sequence of the i-th training text.

6. The method as described in claim 2, characterized in that, The first model to be trained further includes an attention layer, wherein a general matrix is ​​deployed in the side path of the attention layer; the step of adjusting the N training matrices to obtain N auxiliary task matrices with the objective of minimizing the first loss function includes: With the goal of minimizing the first loss function, the general matrix and the N training matrices are adjusted to obtain the adjusted general matrix and the N auxiliary task matrices.

7. The method as described in claim 1, characterized in that, The first feedforward neural network layer includes a first fully connected projection matrix; the i-th auxiliary task training dataset includes the i-th training text and the labeled data of the i-th training text; The first model to be trained is trained using the text task training dataset to obtain N auxiliary task matrices, including: Obtain the input sequence of the first feedforward neural network layer, wherein the input sequence is obtained by encoding the association between any two text positions in the i-th training text; The input sequence is nonlinearly transformed using the first fully connected projection matrix and the i-th training matrix to obtain the feature representation sequence of the i-th training text; Based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, the i-th training matrix is ​​adjusted to obtain the i-th auxiliary task matrix; the auxiliary task processing result of the i-th training text is generated based on the feature representation sequence of the i-th training text.

8. The method as described in claim 7, characterized in that, The first model to be trained further includes an attention layer, wherein a general matrix is ​​deployed in the side path of the attention layer; the step of adjusting the i-th matrix to be trained based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text to obtain the i-th auxiliary task matrix includes: Based on the difference between the auxiliary task processing result of the i-th training text and the labeled data of the i-th training text, the general matrix and the i-th training matrix are adjusted to obtain the adjusted general matrix and the i-th auxiliary task matrix.

9. The method as described in claim 1, characterized in that, The second model to be trained further includes an attention layer; the step of calling the second model to be trained to perform summary generation processing on the sample text to obtain the summary generation result of the sample text includes: The attention layer encodes the associations between different text positions in the sample text to obtain the intermediate sequence corresponding to the sample text; The weighted result corresponding to the sample text is obtained by weighted fusion of the processing results of N auxiliary tasks through the gated fully connected layer; the processing results of the N auxiliary tasks are obtained by feature transformation of the intermediate sequence using the N auxiliary task matrices. The second feedforward neural network layer is invoked to perform a nonlinear transformation on the weighted result corresponding to the sample text to obtain the feature representation sequence of the sample text. Based on the feature representation sequence of the sample text, a summary of the sample text is generated.

10. The method as described in claim 1, characterized in that, The gated fully connected layer includes a second fully connected projection matrix; the process of adjusting the expert matrix based on the difference between the summary generation result and the target summary to obtain the summary generation model includes: Based on the difference between the generated summary and the target summary, a second loss function is constructed; With the goal of minimizing the second loss function, the expert matrix and the second fully connected projection matrix are adjusted to obtain a summary generation model, which includes the adjusted expert matrix and the adjusted second fully connected projection matrix.

11. The method as described in claim 1, characterized in that, The acquisition of the text task training dataset includes: Obtain N training texts and their labeled data; The N training texts and their labeled data are standardized according to the instruction set template to obtain N auxiliary task training datasets. The instruction set template includes a task instruction field, an input field, and a verification field; the value of the task instruction field of the i-th auxiliary task training dataset is generated based on the relationship between the i-th training text and the labeled data of the i-th training text.

12. The method as described in claim 1, characterized in that, Obtain the summary training dataset, including: Obtain M related sub-texts, and concatenate the M sub-texts to obtain the sample text, where M is an integer greater than 1; Obtain the annotation data of the sample text; The sample text and the labeled data are standardized according to the instruction set template to obtain the summary training dataset; The instruction set template includes a task instruction field, an input field, and a verification field; the value of the task instruction field in the summary training dataset is used to describe the summary generation conditions.

13. A method for generating summaries, characterized in that, The method includes: Obtain the intermediate sequence corresponding to the text to be processed, wherein the intermediate sequence is obtained by encoding the association between different text positions in the text to be processed; The results of at least two auxiliary tasks are weighted and fused to obtain the weighted result corresponding to the intermediate sequence; the different auxiliary task processing results are obtained by performing feature transformation on the intermediate sequence using different auxiliary task matrices, and the weights corresponding to each auxiliary task processing result are determined based on the intermediate sequence. A nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed. A summary of the text to be processed is generated based on the feature representation sequence of the text to be processed.

14. The method as described in claim 13, characterized in that, The intermediate sequence is obtained by encoding the association between different text positions in the text to be processed through the attention layer in the summary generation model; the summary generation model also includes a gated fully connected layer, which includes a first fully connected projection matrix and at least two auxiliary task matrices, wherein the at least two auxiliary task matrices are both bypass matrices of the first fully connected projection matrix; The weighted fusion of at least two auxiliary task processing results to obtain the weighted result corresponding to the intermediate sequence includes: The intermediate sequence is projected onto the gate space using the first fully connected projection matrix to obtain the weights corresponding to each auxiliary task matrix. The intermediate sequence is transformed by calling the at least two auxiliary task matrices respectively, resulting in at least two auxiliary task processing results; The processing results of the corresponding auxiliary tasks are weighted and fused according to the weights of each auxiliary task matrix to obtain the weighted result corresponding to the intermediate sequence.

15. The method as described in claim 13, characterized in that, The intermediate sequence is obtained by encoding the association between different text positions in the text to be processed through the attention layer in the summary generation model; the summary generation model also includes a feedforward neural network layer, which includes a second fully connected projection matrix and an expert matrix, wherein the expert matrix is ​​a bypass matrix of the second fully connected projection matrix; The step of performing a nonlinear transformation on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed includes: The weighted result corresponding to the intermediate sequence is nonlinearly transformed using the second fully connected projection matrix to obtain the first transformation result; The weighted result corresponding to the intermediate sequence is linearly transformed using the expert matrix to obtain the second transformation result; The first transformation result is then subjected to nonlinear mapping enhancement processing using the second transformation result to obtain the feature representation sequence of the text to be processed.

16. The method as described in claim 13, characterized in that, The step of obtaining the intermediate sequence corresponding to the text to be processed includes: Obtain the text to be processed, and divide the text to be processed into N sub-sequence blocks according to a preset length, where N is an integer greater than 1; Based on the association between different text positions in each sub-sequence block, the N sub-sequence blocks are encoded to obtain the encoding results of the N sub-sequence blocks; the encoding result of any sub-sequence block is obtained by encoding the association between any two text positions in that sub-sequence block. The encoding results of the N sub-sequence blocks are passed across blocks by a sliding window to obtain the information transmission result of the text to be processed. The encoding results of the N sub-sequence blocks and the information transmission results of the text to be processed are aggregated to obtain the intermediate sequence corresponding to the text to be processed.

17. A data processing apparatus, characterized in that, The data processing device includes: The acquisition unit is used to acquire a text task training dataset and a summary training dataset. The text task training dataset includes N auxiliary task training datasets, and the summary training dataset includes sample text and target summary. The target summary is the labeled data of the sample text, and N is an integer greater than 1. The processing unit is used to train the first model to be trained using the text task training dataset to obtain N auxiliary task matrices. The first model to be trained includes a first feedforward neural network layer. The first feedforward neural network layer has N training matrices deployed in the bypass. The i-th auxiliary task matrix is ​​obtained by training the i-th training matrix using the i-th auxiliary task training dataset. i is a positive integer less than or equal to N. And for calling the second model to be trained to perform summary generation processing on the sample text, to obtain the summary generation result of the sample text; the second model to be trained includes a gated fully connected layer and a second feedforward neural network layer, the N auxiliary task matrices are deployed in the bypass of the gated fully connected layer, and the bypass of the second feedforward neural network layer is deployed with an expert matrix; And to adjust the expert matrix based on the difference between the summary generation result and the target summary, thereby obtaining a summary generation model, which is used to generate text summaries.

18. A summary generation apparatus, characterized in that, The summary generation device includes: An acquisition unit is used to acquire an intermediate sequence corresponding to the text to be processed, wherein the intermediate sequence is obtained by encoding the association between different text positions in the text to be processed; A processing unit is used to perform weighted fusion of the processing results of at least two auxiliary tasks to obtain a weighted result corresponding to the intermediate sequence; the processing results of different auxiliary tasks are obtained by performing feature transformation on the intermediate sequence using different auxiliary task matrices, and the weights corresponding to each processing result of auxiliary tasks are determined based on the intermediate sequence; And a nonlinear transformation is performed on the weighted result corresponding to the intermediate sequence to obtain the feature representation sequence of the text to be processed; And for generating a summary of the text to be processed based on the feature representation sequence of the text to be processed.

19. A computer device, characterized in that, include: A memory, wherein a computer program is stored; A processor is configured to load the computer program to implement the data processing method as described in any one of claims 1-12; or, to load the computer program to implement the summary generation method as described in any one of claims 13-16.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed by the data processing method as described in any one of claims 1-12; or, adapted to be loaded by a processor and executed by the digest generation method as described in any one of claims 13-16.

Images