Cooperative training method and system of language model and medium

Abstract

The invention belongs to the technical field of model training, and discloses a collaborative training method and system of a language model and a medium, and the method comprises the steps: obtaining a sample data set of a power system; the sample data set is input into a pre-training language model for prediction, prediction results are obtained, and the prediction results comprise the executable retrieval probability of the retrieval request obtained based on the request classification task and the prediction probability of the retrieval result obtained based on the retrieval extraction task; constructing a first loss function of the request classification task based on an executable retrieval probability of the retrieval request; constructing a second loss function of the retrieval extraction task based on the prediction probability of the retrieval result; dynamically combining the first loss function and the second loss function to obtain a total loss function; and iteratively updating the parameters of the pre-training language model based on the total loss function to obtain a trained pre-training language model. According to the method, the convergence speed, the prediction precision and the generalization ability of the pre-training language model are improved.

Description

Technical Field

[0001] The present invention belongs to the field of model training technology, and in particular relates to a collaborative training method, system and medium for a language model. Background Art

[0002] A language model is a core concept in the field of natural language processing (NLP). It aims to assign a probability distribution to sentences or sequences in natural language. A language model can predict the probability of a sentence or text, that is, the likelihood of it appearing in natural language.

[0003] In the power sector, language models can be used for power data retrieval, helping users quickly find relevant power system information or historical data, thereby improving the convenience of data retrieval in the power sector.

[0004] However, existing language models still have at least the following flaws when processing power data retrieval tasks: 1. The retrieval semantic information is not comprehensive, and the accuracy of the retrieval results is relatively low; 2. Single-task training methods are often used, and the training efficiency of the model is relatively low. Summary of the Invention

[0005] The purpose of the present invention is to provide a collaborative training method, system and medium for a language model, so as to solve the problems of low training efficiency and low retrieval accuracy of existing language models.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a collaborative training method for a language model, the method comprising: Obtain a sample data set of the power system, wherein the sample data set includes a plurality of text data, a search request, and a true label corresponding to the search request as to whether the corresponding text data can be retrieved; Inputting the sample data set into a pre-trained language model for prediction to obtain prediction results, wherein the pre-trained language model has two training tasks, namely, a request classification task and a retrieval extraction task. The prediction results include: the executable retrieval probability of the retrieval request obtained based on the request classification task and the predicted probability of the retrieval result obtained based on the retrieval extraction task; Constructing a first loss function for the request classification task based on the executable retrieval probability of the retrieval request; Construct the second loss function of the retrieval extraction task based on the predicted probability of the retrieval results; Dynamically combine the first loss function with the second loss function to obtain the total loss function; The parameters of the pre-trained language model are iteratively updated based on the total loss function to obtain a trained pre-trained language model.

[0007] Preferably, the pre-trained language model includes: an encoding layer, a feature extraction layer, a feature fusion layer and a prediction output layer; the output of the encoding layer serves as the input of the feature extraction layer, the first input of the feature fusion layer and the first input of the prediction output layer respectively, the output of the feature extraction layer serves as the second input of the feature fusion layer, and the output of the feature fusion layer serves as the second input of the prediction output layer; wherein, the feature extraction layer is constructed based on the attention mechanism and the gating mechanism, and the feature fusion layer is constructed based on the residual structure; the request classification task and the retrieval extraction task are deployed in the prediction output layer.

[0008] Preferably, the sample data set is input into a pre-trained language model for prediction to obtain a prediction result, including: Encode several text data and retrieval requests based on the encoding layer to obtain the original embedding vector; Based on the attention mechanism of the feature extraction layer, feature extraction is performed on the original embedding vector to obtain a key feature vector, wherein the key feature vector includes: an attention vector of the retrieval request on the text data, an attention vector of the text data on the retrieval request, and an embedding vector of the text data based on the retrieval request; The key feature vectors are interactively screened based on the gating mechanism of the feature extraction layer to obtain the gated output vector; Based on the residual structure of the feature fusion layer, the gated output vector, key feature vector and original embedding vector are fused to obtain a complete semantic vector; Performing retrieval prediction on the complete semantic vector based on the retrieval extraction task of the prediction output layer to obtain the predicted probability of the retrieval result. The predicted probability of the retrieval result is used to represent the probability of each word in the text data being the starting position of the retrieval result and the probability of being the ending position of the retrieval result; Based on the request classification task of the prediction output layer, executable retrieval prediction is performed on the original embedding vector to obtain the executable retrieval probability of the retrieval request. The executable retrieval probability of the retrieval request is used to represent the probability of whether the retrieval request can retrieve the corresponding text data.

[0009] Preferably, the original embedding vector includes: a text vector and a request vector. Based on the attention mechanism of the feature extraction layer, feature extraction is performed on the original embedding vector to obtain a key feature vector, including: Calculate the similarity of the original embedding vector to obtain the similarity matrix; Normalize the similarity matrix to obtain a normalized similarity matrix; Based on the attention mechanism, the normalized similarity matrix and the request vector are subjected to attention calculation to obtain the attention vector of the text data to the retrieval request. Extract the maximum similarity value in each row of the similarity matrix to obtain a column vector; Normalize the column vector to obtain a normalized column vector; Based on the attention mechanism, the normalized column vector and the text vector are subjected to attention calculation to obtain the attention vector of the retrieval request for the text data. The attention vector of the text data on the retrieval request and the attention vector of the retrieval request on the text data are fused to obtain an embedding vector of the text data based on the retrieval request.

[0010] Preferably, the key feature vectors are interactively screened based on the gating mechanism of the feature extraction layer to obtain a gated output vector, including: Based on the gating mechanism, the text vector is concatenated with the attention vector of the text data pair retrieval request to obtain a first concatenated vector; splicing the text vector with the embedding vector of the text data based on the retrieval request based on a gating mechanism to obtain a second spliced ​​vector; The first concatenated vector and the second concatenated vector are averaged to obtain a gated output vector.

[0011] Preferably, the calculation expression of the first loss function is: ; Where, is the first loss function, i is the i-th retrieval request, is the starting position of the search results corresponding to the i-th search request, is the starting position probability vector of the retrieval result corresponding to the i-th retrieval request in the text data. The starting position probability vector is composed of the probabilities of the starting positions corresponding to multiple words. is the end position of the search results corresponding to the i-th search request, is the probability vector of the terminal position of the retrieval result corresponding to the i-th retrieval request in the text data. The terminal position probability vector is composed of the probabilities of the terminal positions corresponding to multiple words. log() is the logarithmic function, and N is the total number of retrieval requests.

[0012] Preferably, the calculation expression of the second loss function is: ; Where, is the second loss function, is the executable retrieval probability of the i-th retrieval request, is the true label of whether the corresponding text data can be retrieved for the i-th retrieval request, log() is the logarithmic function, and N is the total number of retrieval requests.

[0013] Preferably, the first loss function and the second loss function are dynamically combined to obtain a total loss function, including: Constructing a first dynamic weight of the first loss function and a second dynamic weight of the second loss function based on the number of iterations of the iterative update of the pre-trained language model; A total loss function is constructed based on the first loss function and the corresponding first dynamic weight and the second loss function and the corresponding second dynamic weight.

[0014] Preferably, the expression of the total loss function is: ; Where, is the total loss function, is the first dynamic weight of the t-th iteration, is the second dynamic weight of the t-th iteration.

[0015] In a second aspect, the present invention provides a collaborative training system for a language model, for implementing the collaborative training method for a language model, the system comprising: A data acquisition module is used to acquire a sample data set of the power system, wherein the sample data set includes a number of text data, a search request, and a real label corresponding to the search request as to whether the corresponding text data can be retrieved; A retrieval prediction module is used to input a sample data set into a pre-trained language model for prediction to obtain prediction results. The pre-trained language model has two training tasks: a request classification task and a retrieval extraction task. The prediction results include: the executable retrieval probability of the retrieval request obtained based on the request classification task and the predicted probability of the retrieval result obtained based on the retrieval extraction task; A first building module is used to build a first loss function for a request classification task based on an executable retrieval probability of the retrieval request; A second building module is used to build a second loss function for the retrieval extraction task based on the predicted probability of the retrieval result; A third building block is used to dynamically combine the first loss function with the second loss function to obtain a total loss function; The iterative training module is used to iteratively update the parameters of the pre-trained language model based on the total loss function to obtain a trained pre-trained language model.

[0016] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned collaborative training method of the language model when executing the computer program.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned collaborative training method of the language model.

[0018] Beneficial effects: 1. The pre-trained language model of the present invention divides the retrieval task into two training tasks: the request classification task and the retrieval extraction task. The two training tasks are trained simultaneously, which improves the training efficiency of the pre-trained language model. 2. The present invention dynamically combines the first loss function with the second loss function to construct a total loss function. The total loss function can be dynamically adjusted during each training, which can improve the convergence speed of the pre-trained language model while also improving the prediction accuracy and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present invention, but do not constitute a limitation of the embodiments of the present invention. In the accompanying drawings: Figure 1 is a flow chart of a collaborative training method for a language model provided by one embodiment of the present invention; Figure 2 It is a block diagram of a collaborative training system for language models provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly introduced below in conjunction with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0021] Example 1 Figure 1 FIG. 1 is a flow chart of a collaborative training method for a language model provided by an embodiment of the present invention. Figure 1 As shown, this embodiment provides a collaborative training method for a language model, the method comprising: Step S10: Obtain a sample data set of the power system, wherein the sample data set includes a number of text data, a search request, and a real label corresponding to the search request indicating whether the corresponding text data can be retrieved; wherein, a large amount of text data is collected from power system reports, academic papers, operation manuals, etc., and the collected text data is cleaned to remove irrelevant information, correct errors, and unify terminology, etc.; the search request is a search statement entered by the user, such as a simple search statement such as "search for the power generation in xx month of xx year", or a more complex search statement.

[0022] Step S20: Input the sample data set into the pre-trained language model for prediction to obtain a prediction result. The pre-trained language model has two training tasks, namely the request classification task and the retrieval extraction task. The prediction result includes: the executable retrieval probability of the retrieval request obtained based on the request classification task and the predicted probability of the retrieval result obtained based on the retrieval extraction task.

[0023] In this embodiment, the request classification task is used to classify the user's search request to determine whether the search request can retrieve corresponding text data. If the search request can retrieve corresponding text data, the search extraction task is executed to find the corresponding search content from the sample data set. The search content is at least one continuous text segment found, which is used as the search result.

[0024] In this embodiment, the pre-trained language model includes: an encoding layer, a feature extraction layer, a feature fusion layer and a prediction output layer; the output of the encoding layer serves as the input of the feature extraction layer, the first input of the feature fusion layer and the first input of the prediction output layer respectively, the output of the feature extraction layer serves as the second input of the feature fusion layer, and the output of the feature fusion layer serves as the second input of the prediction output layer; wherein, the feature extraction layer is constructed based on the attention mechanism and the gating mechanism, and the feature fusion layer is constructed based on the residual structure; the request classification task and the retrieval extraction task are deployed in the prediction output layer.

[0025] As a further optimization of this embodiment, the sample data set is input into the pre-trained language model for prediction, and the prediction results obtained include: Step S201: Encode a number of text data and search requests based on the encoding layer to obtain an original embedding vector; wherein the original embedding vector includes: a text vector and a request vector; wherein a bag-of-words model can be used to encode the text data and the search request to obtain corresponding text vectors and request vectors.

[0026] Step S202: Based on the attention mechanism of the feature extraction layer, feature extraction is performed on the original embedding vector to obtain a key feature vector, where the key feature vector includes: an attention vector of the retrieval request to the text data, an attention vector of the text data to the retrieval request, and an embedding vector of the text data based on the retrieval request.

[0027] In this embodiment, in order to improve the understanding of text data and search requests and highlight the key parts, this embodiment uses the attention mechanism of the feature extraction layer to extract features from the original embedding vector to obtain key feature vectors. The specific steps are as follows: Step S202.1: Calculate the similarity of the original embedding vector to obtain a similarity matrix. The similarity matrix is ​​calculated as follows: ; Where, is the similarity between the nth word vector of the text data and the mth word vector of the retrieval request, is the nth word vector of the text data, is the mth word vector of the retrieval request, and T is the transpose.

[0028] Step S202.2: normalize the similarity matrix to obtain a normalized similarity matrix.

[0029] Step S202.3: Perform attention calculation on the normalized similarity matrix and the request vector based on the attention mechanism to obtain the attention vector of the text data for the retrieval request.

[0030] Step S202.4: Extract the maximum similarity value in each row of the similarity matrix to obtain a column vector. At this time, the column vector is used to represent which word in the text data is most relevant to a word in the search request. The word in the text data is crucial to the search request.

[0031] Step S202.5: Normalize the column vector to obtain a normalized column vector.

[0032] Step S202.6: Perform attention calculation on the normalized column vector and the text vector based on the attention mechanism to obtain the attention vector of the retrieval request on the text data. The attention vector of the retrieval request on the text data is used to represent the words in the text data related to the words in the retrieval request to highlight the characteristics of the words in the text data.

[0033] Step S202.7: Fuse the attention vector of the text data on the retrieval request and the attention vector of the retrieval request on the text data to obtain an embedding vector of the text data based on the retrieval request.

[0034] In this embodiment, by constructing the attention vector of text data to retrieval request and the attention vector of retrieval request to text data, and then fusing the attention vector of text data to retrieval request and the attention vector of retrieval request to text data, the understanding of text data and retrieval request can be fully deepened.

[0035] Step S203: interactively screening the key feature vectors based on the gating mechanism of the feature extraction layer to obtain a gated output vector.

[0036] In this embodiment, a gating mechanism is used to interactively filter key feature vectors to determine which words in the text data are critical to the search results of the query and which words can be ignored. This helps avoid information redundancy, allowing the model to focus more on important information, thereby improving prediction accuracy. Furthermore, the gating mechanism enhances the robustness of the model, enabling it to maintain good performance even in the presence of noisy or irrelevant information.

[0037] Specifically, the gating mechanism of the feature extraction layer is used to interactively screen key feature vectors to obtain gated output vectors, including: Step S203.1: Based on the gating mechanism, the text vector and the attention vector of the text data search request are concatenated to obtain a first concatenated vector. The function expression of the first concatenated vector is: ; Where, is the first splicing vector, is the first weight, is the attention vector of the text data pair retrieval request, is a text vector.

[0038] Among them, the expression of the first weight is: ; Where, is the activation function, is the first learnable weight matrix, is the first bias term.

[0039] Step S203.2: Based on the gating mechanism, the text vector is concatenated with the embedding vector of the text data based on the search request to obtain a second concatenated vector. The function expression of the first concatenated vector is: ; Where, is the second splicing vector, is the second weight, is the embedding vector of the text data based on the retrieval request, is a text vector.

[0040] Among them, the expression of the second weight is: ; Where, is the activation function, is the second learnable weight matrix, is the second bias term.

[0041] Step S203.3: Perform averaging processing on the first splicing vector and the second splicing vector to obtain a gated output vector.

[0042] In this embodiment, the expression of the gate output vector G is: ; Where mean() is the mean value function.

[0043] Step S204: Based on the residual structure of the feature fusion layer, the gated output vector, the key feature vector and the original embedding vector are fused to obtain a complete semantic vector.

[0044] In this embodiment, the residual structure fuses raw semantic information (i.e., the original embedding vector) with the fine-grained representation obtained through the attention mechanism (i.e., the key feature vector) and the filtered representation obtained through the gating mechanism (i.e., the gated output vector). This helps the model understand the article and question at both coarse and fine granularity, thereby more comprehensively capturing semantic information. Furthermore, by directly passing raw semantic information to subsequent network layers, it effectively alleviates network degradation and enables the model to build deeper neural networks.

[0045] Step S205: Perform retrieval prediction on the complete semantic vector based on the retrieval extraction task of the prediction output layer to obtain the predicted probability of the retrieval result. The predicted probability of the retrieval result is used to represent the probability of each word in the text data being the starting position of the retrieval result and the probability of being the ending position of the retrieval result.

[0046] In this embodiment, the prediction output layer is a fully connected layer. By inputting the complete semantic vector into the fully connected layer, the probability of each word in the text data being the starting position of the retrieval result and the probability of being the ending position of the retrieval result can be obtained.

[0047] Step S206: Based on the request classification task of the prediction output layer, the original embedding vector is subjected to executable retrieval prediction to obtain the executable retrieval probability of the retrieval request, where the executable retrieval probability of the retrieval request is used to represent the probability of whether the retrieval request can retrieve the corresponding text data.

[0048] Step S30: constructing a first loss function for the request classification task based on the executable retrieval probability of the retrieval request.

[0049] In this embodiment, the calculation expression of the first loss function is: ; Where, is the first loss function, i is the i-th retrieval request, is the starting position of the search results corresponding to the i-th search request, is the starting position probability vector of the retrieval result corresponding to the i-th retrieval request in the text data. The starting position probability vector is composed of the probabilities of the starting positions corresponding to multiple words. is the end position of the search results corresponding to the i-th search request, is the probability vector of the terminal position of the retrieval result corresponding to the i-th retrieval request in the text data. The terminal position probability vector is composed of the probabilities of the terminal positions corresponding to multiple words. log() is the logarithmic function, and N is the total number of retrieval requests.

[0050] Step S40: Constructing a second loss function for the retrieval extraction task based on the predicted probability of the retrieval results.

[0051] In this embodiment, the calculation expression of the second loss function is: ; Where, is the second loss function, is the executable retrieval probability of the i-th retrieval request, is the true label of whether the corresponding text data can be retrieved for the i-th retrieval request, log() is the logarithmic function, and N is the total number of retrieval requests.

[0052] Step S50: Dynamically combine the first loss function and the second loss function to obtain a total loss function.

[0053] As a further optimization of this embodiment, the first loss function and the second loss function are dynamically combined to obtain a total loss function, including: Step S501: constructing a first dynamic weight of a first loss function and a second dynamic weight of a second loss function based on the number of iterations of the iterative update of the pre-trained language model.

[0054] Step S502: constructing a total loss function based on the first loss function and the corresponding first dynamic weight and the second loss function and the corresponding second dynamic weight.

[0055] Among them, the expression of the total loss function is: ; Where, is the total loss function, is the first dynamic weight of the t-th iteration, is the second dynamic weight of the t-th iteration.

[0056] In this embodiment, the loss functions of each related task form a total loss function, thereby achieving rapid convergence for the entire prediction model. The contribution of each related task loss function to the total loss function has a direct impact on the model's convergence speed, prediction accuracy, and generalization ability. Existing methods for constructing a total loss function linearly sum each loss function according to its proportion (weight). However, the performance of weighted linear summation methods during model training is highly dependent on manually set weights.

[0057] In this embodiment, the first dynamic weight and the second dynamic weight can automatically find the optimal value as the number of iterations changes, which greatly saves time and improves the operating efficiency of the prediction model, thereby improving the prediction accuracy and generalization ability of the model. The first dynamic weight and the second dynamic weight can be solved using a particle swarm algorithm or a genetic algorithm.

[0058] Step S60: Iteratively update the parameters of the pre-trained language model based on the total loss function to obtain a trained pre-trained language model.

[0059] The pre-trained language model of the present invention divides the retrieval task into two training tasks: a request classification task and a retrieval extraction task. The two training tasks are trained simultaneously, which improves the training efficiency of the pre-trained language model. Secondly, the first loss function and the second loss function are dynamically combined to construct a total loss function. The total loss function can be dynamically adjusted during each training, which can improve the convergence speed of the pre-trained language model, while also improving the prediction accuracy and generalization ability of the model.

[0060] Example 2 Figure 2 FIG. 1 is a block diagram of a collaborative training system for a language model provided by an embodiment of the present invention. Figure 2 As shown, this embodiment provides a collaborative training system for a language model, which is used to implement the collaborative training method for a language model in Example 1. The system includes: A data acquisition module is used to acquire a sample data set of the power system, wherein the sample data set includes a number of text data, a search request, and a real label corresponding to the search request as to whether the corresponding text data can be retrieved; A retrieval prediction module is used to input a sample data set into a pre-trained language model for prediction to obtain prediction results. The pre-trained language model has two training tasks: a request classification task and a retrieval extraction task. The prediction results include: the executable retrieval probability of the retrieval request obtained based on the request classification task and the predicted probability of the retrieval result obtained based on the retrieval extraction task; A first building module is used to build a first loss function for a request classification task based on an executable retrieval probability of the retrieval request; A second building module is used to build a second loss function for the retrieval extraction task based on the predicted probability of the retrieval result; A third building block is used to dynamically combine the first loss function with the second loss function to obtain a total loss function; The iterative training module is used to iteratively update the parameters of the pre-trained language model based on the total loss function to obtain a trained pre-trained language model.

[0061] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the collaborative training method of the language model in the first embodiment is implemented.

[0062] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the collaborative training method of the language model in the first embodiment is implemented.

[0063] The pre-trained language model of the present invention divides the retrieval task into two training tasks: a request classification task and a retrieval extraction task. The two training tasks are trained simultaneously, which improves the training efficiency of the pre-trained language model. Secondly, the first loss function and the second loss function are dynamically combined to construct a total loss function. The total loss function can be dynamically adjusted during each training, which can improve the convergence speed of the pre-trained language model, while also improving the prediction accuracy and generalization ability of the model.

[0064] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0065] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or boxes.

[0066] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A collaborative training method for a language model, characterized in that: The method comprises: Obtain a sample data set of the power system, wherein the sample data set includes a plurality of text data, a search request, and a true label corresponding to the search request as to whether the corresponding text data can be retrieved; Inputting the sample data set into a pre-trained language model for prediction to obtain prediction results, wherein the pre-trained language model has two training tasks, namely, a request classification task and a retrieval extraction task. The prediction results include: the executable retrieval probability of the retrieval request obtained based on the request classification task and the predicted probability of the retrieval result obtained based on the retrieval extraction task; Constructing a first loss function for the request classification task based on the executable retrieval probability of the retrieval request; Construct the second loss function of the retrieval extraction task based on the predicted probability of the retrieval results; Dynamically combine the first loss function with the second loss function to obtain the total loss function; The parameters of the pre-trained language model are iteratively updated based on the total loss function to obtain a trained pre-trained language model.

2. The collaborative training method of a language model according to claim 1, characterized in that The pre-trained language model includes: an encoding layer, a feature extraction layer, a feature fusion layer and a prediction output layer; the output of the encoding layer serves as the input of the feature extraction layer, the first input of the feature fusion layer and the first input of the prediction output layer respectively, the output of the feature extraction layer serves as the second input of the feature fusion layer, and the output of the feature fusion layer serves as the second input of the prediction output layer; wherein, the feature extraction layer is constructed based on the attention mechanism and the gating mechanism, and the feature fusion layer is constructed based on the residual structure; the request classification task and the retrieval extraction task are deployed in the prediction output layer.

3. The collaborative training method of a language model according to claim 2, characterized in that: Input the sample dataset into the pre-trained language model for prediction and obtain the prediction results, including: Encode several text data and retrieval requests based on the encoding layer to obtain the original embedding vector; Based on the attention mechanism of the feature extraction layer, feature extraction is performed on the original embedding vector to obtain a key feature vector, wherein the key feature vector includes: an attention vector of the retrieval request on the text data, an attention vector of the text data on the retrieval request, and an embedding vector of the text data based on the retrieval request; The key feature vectors are interactively screened based on the gating mechanism of the feature extraction layer to obtain the gated output vector; Based on the residual structure of the feature fusion layer, the gated output vector, key feature vector and original embedding vector are fused to obtain a complete semantic vector; Performing retrieval prediction on the complete semantic vector based on the retrieval extraction task of the prediction output layer to obtain the predicted probability of the retrieval result. The predicted probability of the retrieval result is used to represent the probability of each word in the text data being the starting position of the retrieval result and the probability of being the ending position of the retrieval result; Based on the request classification task of the prediction output layer, executable retrieval prediction is performed on the original embedding vector to obtain the executable retrieval probability of the retrieval request. The executable retrieval probability of the retrieval request is used to represent the probability of whether the retrieval request can retrieve the corresponding text data.

4. The collaborative training method of a language model according to claim 3, characterized in that: The original embedding vector includes: a text vector and a request vector. Based on the attention mechanism of the feature extraction layer, feature extraction is performed on the original embedding vector to obtain a key feature vector, including: Calculate the similarity of the original embedding vector to obtain the similarity matrix; Normalize the similarity matrix to obtain a normalized similarity matrix; Based on the attention mechanism, the normalized similarity matrix and the request vector are subjected to attention calculation to obtain the attention vector of the text data to the retrieval request. Extract the maximum similarity value in each row of the similarity matrix to obtain a column vector; Normalize the column vector to obtain a normalized column vector; Based on the attention mechanism, the normalized column vector and the text vector are subjected to attention calculation to obtain the attention vector of the retrieval request for the text data. The attention vector of the text data on the retrieval request and the attention vector of the retrieval request on the text data are fused to obtain an embedding vector of the text data based on the retrieval request.

5. The collaborative training method of a language model according to claim 4, characterized in that: Based on the gating mechanism of the feature extraction layer, key feature vectors are interactively screened to obtain gated output vectors, including: Based on the gating mechanism, the text vector is concatenated with the attention vector of the text data pair retrieval request to obtain a first concatenated vector; splicing the text vector with the embedding vector of the text data based on the retrieval request based on a gating mechanism to obtain a second spliced ​​vector; The first concatenated vector and the second concatenated vector are averaged to obtain a gated output vector.

6. The collaborative training method of a language model according to claim 4, characterized in that: The calculation expression of the first loss function is: ； Where, is the first loss function, i is the i-th retrieval request, is the starting position of the search results corresponding to the i-th search request, is the starting position probability vector of the retrieval result corresponding to the i-th retrieval request in the text data. The starting position probability vector is composed of the probabilities of the starting positions corresponding to multiple words. is the end position of the search results corresponding to the i-th search request, is the probability vector of the terminal position of the retrieval result corresponding to the i-th retrieval request in the text data. The terminal position probability vector is composed of the probabilities of the terminal positions corresponding to multiple words. log() is the logarithmic function, and N is the total number of retrieval requests.

7. The collaborative training method of a language model according to claim 6, characterized in that: The calculation expression of the second loss function is: ； Where, is the second loss function, is the executable retrieval probability of the i-th retrieval request, is the true label of whether the corresponding text data can be retrieved for the i-th retrieval request, log() is the logarithmic function, and N is the total number of retrieval requests.

8. The collaborative training method of a language model according to claim 7, characterized in that: The first loss function is dynamically combined with the second loss function to obtain the total loss function, including: Constructing a first dynamic weight of the first loss function and a second dynamic weight of the second loss function based on the number of iterations of the iterative update of the pre-trained language model; A total loss function is constructed based on the first loss function and the corresponding first dynamic weight and the second loss function and the corresponding second dynamic weight.

9. The collaborative training method of a language model according to claim 8, characterized in that: The expression of the total loss function is: ； Where, is the total loss function, is the first dynamic weight of the t-th iteration, is the second dynamic weight of the t-th iteration.

10. A language model collaborative training system, used to implement the language model collaborative training method according to any one of claims 1 to 9, characterized in that: The system comprises: A data acquisition module is used to acquire a sample data set of the power system, wherein the sample data set includes a number of text data, a search request, and a real label corresponding to the search request as to whether the corresponding text data can be retrieved; A retrieval prediction module is used to input a sample data set into a pre-trained language model for prediction to obtain prediction results. The pre-trained language model has two training tasks: a request classification task and a retrieval extraction task. The prediction results include: the executable retrieval probability of the retrieval request obtained based on the request classification task and the predicted probability of the retrieval result obtained based on the retrieval extraction task; A first building module is used to build a first loss function for a request classification task based on an executable retrieval probability of the retrieval request; A second building module is used to build a second loss function for the retrieval extraction task based on the predicted probability of the retrieval result; A third building block is used to dynamically combine the first loss function with the second loss function to obtain a total loss function; The iterative training module is used to iteratively update the parameters of the pre-trained language model based on the total loss function to obtain a trained pre-trained language model.

Images