Parameter setting method, text generation method, device, equipment and program product

Abstract

The invention provides a parameter setting method and device, a text generation method and device, equipment and a program product, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining a pre-training language model and a preset low-rank conversion parameter; for network layers corresponding to the same network layer type in the pre-training language model, setting a preset low-rank conversion parameter as a low-rank conversion parameter of a top network layer, and determining the similarity between adjacent network layers; if the similarity is not less than a preset threshold value, taking the low-rank conversion parameter of the upper network layer in the adjacent network layers as the low-rank conversion parameter of the lower network layer in the adjacent network layers; if the similarity is smaller than a preset threshold value, determining an adjustment coefficient according to the similarity, and increasing a low-rank conversion parameter of an upper network layer in the adjacent network layers by using the adjustment coefficient to obtain a low-rank conversion parameter of a lower network layer in the adjacent network layers; the low-rank conversion parameters can be adaptively set for the network layer so as to improve the fine tuning training effect of the model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a parameter setting method, a text generation method, a device, a equipment, and a program product. Background Art

[0002] With the continuous development of artificial intelligence technology, large-scale pre-trained language models are gradually being applied in many fields. To ensure that pre-trained language models can adapt to different field tasks, a common model training method is to pre-train a basic language model on a huge general data set, and then apply it to downstream field tasks through fine-tuning technology. However, the huge number of model parameters brings great challenges to the subsequent fine-tuning process, such as resource requirements and training time.

[0003] In related technologies, in order to reduce resource consumption of the model during fine-tuning and improve fine-tuning efficiency, a low-rank approximation method can be used to perform a low-rank transformation on the weight matrix of the model network layer, and use multiple matrices of lower dimensions to approximate the parameter changes of the original weights. However, when performing a low-rank transformation on the weight matrix of the model network layer, related technologies usually use fixed low-rank transformation parameters for processing, which can easily lead to redundancy of shallow network parameters or serious information loss in deep networks, thereby affecting the efficiency and performance of model fine-tuning. Summary of the invention

[0004] The purpose of the present invention is to provide a parameter setting method, a text generation method, an apparatus, a device, and a program product, which can adaptively set low-rank transformation parameters for the network layer in a pre-trained language model, avoid using fixed low-rank transformation parameters, and thus improve the model fine-tuning training effect.

[0005] In order to solve the above technical problems, the present invention provides a parameter setting method, comprising:

[0006] Get the pre-trained language model and preset low-rank transformation parameters;

[0007] For network layers corresponding to the same network layer type in the pre-trained language model, setting the preset low-rank transformation parameters as low-rank transformation parameters of the top network layer, and determining the similarity between adjacent network layers;

[0008] If the similarity is not less than a preset threshold, using the low-rank transformation parameters of the upper network layer in the adjacent network layers as the low-rank transformation parameters of the lower network layer in the adjacent network layers;

[0009] If the similarity is less than the preset threshold, an adjustment coefficient is determined according to the similarity, and the low-rank transformation parameter of the upper network layer in the adjacent network layer is increased by using the adjustment coefficient to obtain the low-rank transformation parameter of the lower network layer in the adjacent network layer.

[0010] Optionally, determining an adjustment coefficient according to the similarity, and increasing a low-rank transformation parameter of an upper network layer in the adjacent network layer by using the adjustment coefficient, includes:

[0011] Determining a correlation factor according to the similarity; wherein the similarity is positively correlated with the correlation factor;

[0012] The reciprocal of the correlation factor is used as the adjustment coefficient, and the low-rank transformation parameter of the upper network layer in the adjacent network layer is increased by using the adjustment coefficient.

[0013] Optionally, obtaining preset low-rank transformation parameters includes:

[0014] Determine the network layer type corresponding to each network layer in the pre-trained language model;

[0015] Obtain preset low-rank transformation parameters corresponding to each of the network layer types.

[0016] Optionally, for network layers corresponding to the same network layer type in the pre-trained language model, setting the preset low-rank conversion parameter as a low-rank conversion parameter of the top network layer includes:

[0017] For network layers corresponding to the same network layer type in the pre-trained language model, the preset low-rank transformation parameters corresponding to the network layer type are set as low-rank transformation parameters of the top network layer corresponding to the network layer type.

[0018] Optionally, it also includes:

[0019] Creating different preset low-rank transformation parameter groups; wherein the preset low-rank transformation parameter groups store the preset low-rank transformation parameters corresponding to each of the network layer types, and different preset low-rank transformation parameter groups store different preset low-rank transformation parameters;

[0020] Determining different low-rank transformation parameters for the network layer in the pre-trained language model based on the different preset low-rank transformation parameter groups;

[0021] Fine-tune each network layer in the pre-trained language model based on different low-rank transformation parameters to determine model performance values ​​corresponding to different preset low-rank transformation parameter groups;

[0022] An optimal preset low-rank transformation parameter set is determined based on the model performance value.

[0023] Optionally, determining similarities between adjacent network layers includes:

[0024] Determine the characteristic distance between the weight matrices of the adjacent network layers, and use the characteristic distance as the similarity.

[0025] Optionally, determining the characteristic distance between the weight matrices of the adjacent network layers includes:

[0026] Performing singular value decomposition on a first weight matrix of an upper network layer in an adjacent network layer to obtain a first non-negative diagonal matrix and a first right singular matrix, and determining the rank of the first non-negative diagonal matrix;

[0027] Determine a first screening quantity according to the rank of the first non-negative diagonal matrix, and screen the first right singular matrix for the first singular vectors whose number is the first screening quantity;

[0028] Performing singular value decomposition on a second weight matrix corresponding to a lower network layer in an adjacent network layer to obtain a second non-negative diagonal matrix and a second right singular matrix, and determining the rank of the second non-negative diagonal matrix;

[0029] Determine a second screening number according to the rank of the second non-negative diagonal matrix, and screen the first second singular vectors whose number is the second screening number in the second right singular matrix;

[0030] The similarity is determined using the first singular vectors, the rank of the first non-negative diagonal matrix, the second singular vectors, and the rank of the second non-negative diagonal matrix.

[0031] Optionally, the pre-trained language model is composed of multiple transformer network layers connected in series.

[0032] Optionally, it also includes:

[0033] Performing matrix decomposition on the weight matrix of the network layer according to the low-rank transformation parameter to obtain a low-rank transformation weight matrix corresponding to the network layer;

[0034] The low-rank transformation weight matrix of the network layer is fine-tuned and trained.

[0035] Optionally, fine-tuning the low-rank transformation weight matrix of the network layer includes:

[0036] Get the preset training text;

[0037] The preset training text is processed layer by layer using each network layer in the pre-trained language model to obtain model output data; wherein the output of the network layer is fused output data obtained by fusion processing of the first output data and the second output data, the first output data is the processing result of the preset training text by the weight matrix of the network layer, and the second output data is the processing result of the preset training text by the low-rank transformation weight matrix of the network layer;

[0038] A loss value is determined based on the model output data, and parameters of the low-rank transformation weight matrix of each network layer are updated according to the loss value.

[0039] Optionally, the preset training text is processed layer by layer using each network layer in the pre-trained language model to obtain model output data, including:

[0040] Using the first network layer in the pre-trained language model as the current network layer, using the preset training text as the data to be processed, and inputting the data to be processed into the current network layer;

[0041] Processing the data to be processed using the weight matrix of the current network layer to obtain the first output data;

[0042] Using the low-rank transformation weight matrix of the current network layer to process the data to be processed to obtain the second output data, and using the scaling parameter to scale the second output data;

[0043] Superimposing the first output data with the second output data after scaling processing to obtain the fused output data;

[0044] Determining whether there is a lower network layer that has not processed the preset training text;

[0045] If yes, the fused output data of the current network layer is used as the data to be processed, the current network layer is updated to the lower network layer of the current network layer, and the step of inputting the data to be processed into the current network layer is entered;

[0046] If not, the fused output data of the current network layer is used as the model output data.

[0047] Optionally, before scaling the second output data using the scaling parameter, the method further includes:

[0048] Determine a scaling parameter corresponding to the current network layer by using a low-rank transformation parameter of the current network layer and a preset scaling factor;

[0049] The scaling process of the second output data by using the scaling parameter includes:

[0050] The second output data is scaled using a scaling parameter corresponding to the current network layer.

[0051] The present invention also provides a text generation method, comprising:

[0052] Obtaining user input text and a pre-trained language model; wherein the network layer in the pre-trained language model comprises a weight matrix and a low-rank conversion weight matrix, the low-rank conversion weight matrix is ​​generated using the weight matrix and the low-rank conversion parameters corresponding to the network layer, and the low-rank conversion parameters are generated according to the parameter setting method as described above;

[0053] Using each network layer in the pre-trained language model to process the user input text layer by layer to obtain model output data; wherein the output of the network layer is fused output data obtained by fusion processing of the first output data and the second output data, the first output data is the result of the weight matrix of the network layer processing the user input text, and the second output data is the result of the low-rank transformation weight matrix of the network layer processing the user input text;

[0054] Text generation is performed based on the model output data.

[0055] Optionally, the user input text is processed layer by layer using each network layer in the pre-trained language model to obtain model output data, including:

[0056] Using the first network layer in the pre-trained language model as the current network layer, using the user input text as data to be processed, and inputting the data to be processed into the current network layer;

[0057] Processing the data to be processed using the weight matrix of the current network layer to obtain the first output data;

[0058] Using the low-rank transformation weight matrix of the current network layer to process the data to be processed to obtain the second output data, and using the scaling parameter to scale the second output data;

[0059] Superimposing the first output data with the second output data after scaling processing to obtain the fused output data;

[0060] Determining whether there is a lower network layer that has not processed the user input text;

[0061] If yes, the fused output data of the current network layer is used as the data to be processed, the current network layer is updated to the lower network layer of the current network layer, and the step of inputting the data to be processed into the current network layer is entered;

[0062] If not, the fused output data of the current network layer is used as the model output data.

[0063] Optionally, before scaling the second output data using the scaling parameter, the method further includes:

[0064] Determine a scaling parameter corresponding to the current network layer by using a low-rank transformation parameter of the current network layer and a preset scaling factor;

[0065] The scaling process of the second output data by using the scaling parameter includes:

[0066] The second output data is scaled using a scaling parameter corresponding to the current network layer.

[0067] The present invention also provides a parameter setting device, comprising:

[0068] An acquisition module is used to obtain a pre-trained language model and preset low-rank conversion parameters;

[0069] A network layer processing module, for setting the preset low-rank transformation parameters as low-rank transformation parameters of the top network layer for the network layers corresponding to the same network layer type in the pre-trained language model, and determining the similarity between adjacent network layers;

[0070] A first low-rank conversion parameter setting module, configured to use the low-rank conversion parameters of the upper network layer in the adjacent network layers as the low-rank conversion parameters of the lower network layer in the adjacent network layers if the similarity is not less than a preset threshold;

[0071] The second low-rank conversion parameter setting module is used to determine the adjustment coefficient according to the similarity if the similarity is less than the preset threshold, and use the adjustment coefficient to increase the low-rank conversion parameter of the upper network layer in the adjacent network layer to obtain the low-rank conversion parameter of the lower network layer in the adjacent network layer.

[0072] The present invention also provides a text generation device, comprising:

[0073] An acquisition module, used to acquire user input text and a pre-trained language model; wherein the network layer in the pre-trained language model comprises a weight matrix and a low-rank conversion weight matrix, wherein the low-rank conversion weight matrix is ​​generated using the weight matrix and the low-rank conversion parameters corresponding to the network layer, and the low-rank conversion parameters are generated according to the parameter setting method as described above;

[0074] A model processing module, used to process the user input text layer by layer using each network layer in the pre-trained language model to obtain model output data; wherein the output of each network layer is fused output data obtained by fusion processing of first output data and second output data, the first output data is the processing result of the user input text by the weight matrix of the network layer, and the second output data is the processing result of the user input text by the low-rank transformation weight matrix of the network layer;

[0075] A text generation module is used to generate text based on the model output data.

[0076] The present invention also provides an electronic device, comprising:

[0077] Memory for storing computer programs;

[0078] A processor is used to implement the parameter setting method or the text generation method as described above when executing the computer program.

[0079] The present invention also provides a computer program product, including a computer program or instructions, which implement the parameter setting method or the text generation method as described above when executed by a processor.

[0080] The present invention also provides a non-volatile computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the parameter setting method or the text generation method as described above is implemented.

[0081] The present invention provides a parameter setting method, comprising: obtaining a pre-trained language model and preset low-rank conversion parameters; for network layers corresponding to the same network layer type in the pre-trained language model, setting the preset low-rank conversion parameters as low-rank conversion parameters of the top network layer, and determining the similarity between adjacent network layers; if the similarity is not less than a preset threshold, using the low-rank conversion parameters of the upper network layer in the adjacent network layers as the low-rank conversion parameters of the lower network layer in the adjacent network layers; if the similarity is less than the preset threshold, determining an adjustment coefficient according to the similarity, and using the adjustment coefficient to increase the low-rank conversion parameters of the upper network layer in the adjacent network layers to obtain the low-rank conversion parameters of the lower network layer in the adjacent network layers.

[0082] The beneficial effects of the present invention are as follows: the present invention can first obtain a pre-trained language model and preset low-rank conversion parameters, and for network layers corresponding to the same type of network layers in the pre-trained language model, the preset low-rank conversion parameters can be set as the low-rank conversion parameters of the top network layer, and the similarity between adjacent network layers is determined. Subsequently, the present invention determines whether the similarity between adjacent network layers is less than a preset threshold. If the similarity is not less than the preset threshold, it means that the pair of adjacent network layers is relatively similar, and the same low-rank conversion parameters can be used for low-rank conversion, and then the low-rank conversion parameters of the upper network layer in the adjacent network layer can be used as the low-rank conversion parameters of the lower network layer in the adjacent network layer; if the similarity is less than the preset threshold, it means that the pair of adjacent network layers are not similar, and a larger low-rank conversion parameter needs to be set for the lower network layer to retain information, and then the adjustment coefficient can be determined according to the similarity, and the low-rank conversion parameters of the upper network layer in the adjacent network layer are increased by using the adjustment coefficient to obtain the low-rank conversion parameters of the lower network layer in the adjacent network layer. In this way, the present invention can adaptively set low-rank conversion parameters for the lower network layer according to the similarity between adjacent network layers, and generally ensure that the deeper the network layer, the larger the low-rank conversion parameter, which can effectively ensure the low-rank conversion effect of the model and reduce the problem of information loss in the deep network, thereby improving the efficiency and performance of model fine-tuning training. The present invention also provides a text generation method, a parameter setting device, a text generation device, an electronic device, a computer program product, and a computer-readable storage medium, which have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0084] Figure 1 A flow chart of a parameter setting method provided by an embodiment of the present invention;

[0085] Figure 2 A schematic diagram of matrix decomposition provided by an embodiment of the present invention;

[0086] Figure 3 A flowchart of a text generation method provided by an embodiment of the present invention;

[0087] Figure 4 A schematic diagram of a parameter setting process provided by an embodiment of the present invention;

[0088] Figure 5 A structural block diagram of a parameter setting device provided by an embodiment of the present invention;

[0089] Figure 6 A structural block diagram of a text generation device provided by an embodiment of the present invention;

[0090] Figure 7 The present invention provides a structural block diagram of an electronic device. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0092] A pre-trained language model is a machine learning model that has been trained with a large amount of text data, and is mainly used to parse and generate natural language text. The common application scenario of this model is: receiving question text input by the user, and outputting output text that meets the user's needs based on the question text. With the continuous development of artificial intelligence technology, large-scale pre-trained language models (such as LLM models, Large Language Models, and large language models) have gradually been applied in many fields. To ensure that the pre-trained language model can adapt to different field tasks, the common model training method is to pre-train a basic language model on a huge general data set, and then apply it to downstream field tasks through fine-tuning technology. However, the huge number of model parameters brings great challenges to the subsequent fine-tuning process, such as resource requirements, training time, etc.

[0093] In related technologies, in order to reduce resource consumption of the model during fine-tuning and improve fine-tuning efficiency, a low-rank approximation method can be used to perform a low-rank transformation on the weight matrix of the model network layer, and use multiple matrices of lower dimensions to approximate the parameter changes of the original weights. However, when performing a low-rank transformation on the weight matrix of the model network layer, related technologies usually use fixed low-rank transformation parameters for processing, which can easily lead to redundancy of shallow network parameters or serious information loss in deep networks, thereby affecting the efficiency and performance of model fine-tuning.

[0094] In view of this, in order to solve the technical problem of how to better determine the low-rank transformation parameters for each network layer of the pre-trained language model, the present invention can provide a parameter setting method, which can adaptively set the low-rank transformation parameters for the network layer in the pre-trained language model, avoid using fixed low-rank transformation parameters, and thus improve the model fine-tuning training effect.

[0095] For easier understanding, please refer to Figure 1 , Figure 1 A flow chart of a parameter setting method provided by an embodiment of the present invention, the method may include:

[0096] S101. Obtain a pre-trained language model and preset low-rank conversion parameters.

[0097] In this step, the pre-trained language model refers to a language model that has been trained with a huge amount of general data and needs to be fine-tuned. This embodiment does not limit the specific structure of the pre-trained language model. For example, it can be composed of multiple transformer network layers (Transformer) in series. Among them, the core part of the transformer network layer includes the attention layer (AttentionLayer) and the feed-forward network layer (FFN, Feed-Forward Neural Network), and the feed-forward network layer occupies 70% of the parameters of the model. By superimposing the transformer network layer N times, the model can realize multiple extractions of data features and semantic understanding.

[0098] Furthermore, this step also obtains preset low-rank conversion parameters. Different from the related art, the preset low-rank conversion parameters obtained here are only used as low-rank conversion parameters of the top network layer, and are not used as low-rank conversion parameters of the remaining network layers. The low-rank conversion parameters of the remaining network layers will be adjusted based on the preset low-rank conversion parameters. It should be noted that this embodiment does not limit the specific values ​​of the preset low-rank conversion parameters, which can be set according to actual application requirements.

[0099] S102. For network layers corresponding to the same network layer type in the pre-trained language model, the preset low-rank transformation parameters are set as low-rank transformation parameters of the top network layer, and the similarity between adjacent network layers is determined.

[0100] In this step, low-rank conversion parameters will be set in network layers of the same network layer type. Specifically, in the present embodiment, in network layers of the same network layer type, preset low-rank conversion parameters may be set as low-rank conversion parameters of the top network layer; then, the similarity between adjacent network layers may be determined to set low-rank conversion parameters for each network layer below the top network layer according to the similarity. It should be pointed out here that the "adjacent network layers" here are hierarchically adjacent within the same category, and have a logically adjacent relationship, which are not adjacent in the actual model structure. For example, the present embodiment may classify the network layers in the pre-trained language model as follows:

[0101] Table 1 Example of low-rank approximate fine-tuning module list

[0102]

[0103] For category c1, there are N module sequences, called modules i c1 and module (i+1) c1 They are adjacent network layers, but they are not adjacent in the overall model structure because there will be other types of network layers in the middle.

[0104] Furthermore, since the essence of low-rank conversion is to compress the weight matrix, and different types of network layers have different uses and importance, different strengths can be adopted for low-rank conversion. To this end, the present embodiment can also set corresponding preset low-rank conversion parameters for different network layers. Specifically, the present embodiment can determine the network layer type corresponding to each network layer in the pre-trained language model for classification; then, the preset low-rank conversion parameters corresponding to each network layer type can be obtained, wherein the preset low-rank conversion parameters corresponding to different network layer types may be different. Furthermore, the present embodiment can set the preset low-rank conversion parameters corresponding to the network layer type as the low-rank conversion parameters of the top network layer corresponding to the network layer type, so that different strengths can be adopted to perform low-rank conversion on network layers of different network layer types.

[0105] Based on this, obtaining the preset low-rank transformation parameters may include:

[0106] Step 11: Determine the network layer type corresponding to each network layer in the pre-trained language model;

[0107] Step 12: Obtain preset low-rank transformation parameters corresponding to each of the network layer types.

[0108] Based on this, for the network layers corresponding to the same network layer type in the pre-trained language model, the preset low-rank conversion parameters are set as the low-rank conversion parameters of the top network layer, including:

[0109] Step 21: For the network layers corresponding to the same network layer type in the pre-trained language model, set the preset low-rank conversion parameters corresponding to the network layer type as the low-rank conversion parameters of the top network layer corresponding to the network layer type.

[0110] Furthermore, this embodiment does not limit how to calculate the similarity between network layers. For example, the characteristic distance between the weight matrices of adjacent network layers may be determined, and the characteristic distance may be used as the similarity.

[0111] Based on this, determining the similarity between adjacent network layers may include:

[0112] Step 31: Determine the characteristic distance between the weight matrices of the adjacent network layers, and use the characteristic distance as the similarity.

[0113] It should be noted that this embodiment does not limit the specific feature distance type, and can be set according to actual application requirements. For example, Euclidean distance, Chebyshev distance, Manhattan distance, Minkowski distance, Mahalanobis distance, symmetric point distance, cosine similarity, etc. can be adopted. In this embodiment, in order to effectively indicate the similarity relationship between subspaces corresponding to the intrinsic rank of the weight matrix, the Grassmann distance between the weight matrices can be calculated, and the Grassmann distance is used as the similarity between adjacent network layers. The specific calculation process of Grassmann is introduced below.

[0114] Based on this, determining the characteristic distance between the weight matrices of the adjacent network layers may include:

[0115] Step 41: Perform singular value decomposition on the first weight matrix of the upper network layer in the adjacent network layer to obtain a first non-negative diagonal matrix and a first right singular matrix, and determine the rank of the first non-negative diagonal matrix.

[0116] Specifically, the first weight matrix M 1 Perform singular value decomposition SVD (M 1 )= ,in denotes the first nonnegative diagonal matrix, and V denotes the first right singular matrix.

[0117] Step 42: Determine a first screening quantity according to the rank of the first non-negative diagonal matrix, and screen the first right singular matrix for the first singular vectors whose number is the first screening quantity.

[0118] In this step, you can The rank of the first right singular matrix is ​​screened top i dimensional data (i.e., the first singular vector), i is The rank of the first weight matrix contains the most abundant information in the top i In a dimension.

[0119] Step 43: Perform singular value decomposition on the second weight matrix corresponding to the lower network layer in the adjacent network layer to obtain a second non-negative diagonal matrix and a second right singular matrix, and determine the rank of the second non-negative diagonal matrix.

[0120] Step 44: Determine a second screening quantity according to the rank of the second non-negative diagonal matrix, and screen the first second singular vectors whose number is the second screening quantity in the second right singular matrix.

[0121] Steps 43 and 44 are similar to steps 41 and 42.

[0122] Step 45: Determine the similarity using the first singular vectors, the rank of the first non-negative diagonal matrix, the second singular vectors, and the rank of the second non-negative diagonal matrix.

[0123] Specifically, the similarity between the weight matrices of adjacent network layers can be expressed as:

[0124] ;

[0125] Where sim represents similarity. When sim=0, the two operators (i.e., weight matrices) are orthogonal; when sim=1, the two operators express the same subspace, and the similarity range is [0,1]. GD represents Grassmann distance, M 1 and M 2 Represents two weight matrices. Represents the matrix M 1 The top of the right singular matrix i dimensional data, if the matrix M 1 Perform singular value decomposition SVD (M 1 )= , then top k That is equal to rank. The rank of the matrix M 1 The highest dimension value where the information represented is located, that is, M 1 The most informative content is contained in the top i In the same way is the matrix M 2 The top of the right singular matrix j Dimensional data.

[0126] S103: Determine whether the similarity is less than a preset threshold.

[0127] As described above, this embodiment sets low-rank conversion parameters for each network layer below the top network layer according to the similarity. In this step, it can be determined whether adjacent network layers are similar based on the relationship between the similarity and the preset threshold, and different low-rank conversion parameter setting methods can be adopted in the two cases of similarity and dissimilarity. It should be noted that this embodiment does not limit the specific value of the preset threshold, which can be set according to actual application requirements.

[0128] S104: If the similarity is not less than a preset threshold, the low-rank transformation parameters of the upper network layer in the adjacent network layers are used as the low-rank transformation parameters of the lower network layer in the adjacent network layers.

[0129] In this step, if it is determined that the similarity between adjacent network layers is not less than a preset threshold, it can be determined that the two network layers are relatively similar, and then the subspaces described by the weight matrices of the two network layers are relatively close, and the same low-rank transformation parameters can be used for low-rank transformation. Therefore, in this step, the low-rank transformation parameters of the upper network layer in the adjacent network layers can be directly used as the low-rank transformation parameters of the lower network layer in the adjacent network layers.

[0130] S105. If the similarity is less than the preset threshold, an adjustment coefficient is determined according to the similarity, and the low-rank transformation parameter of the upper network layer in the adjacent network layer is increased by using the adjustment coefficient to obtain the low-rank transformation parameter of the lower network layer in the adjacent network layer.

[0131] In this step, if it is determined that the similarity between adjacent network layers is less than a preset threshold, it can be determined that the two network layers are not similar, and thus the subspaces described by the weight matrices of the two network layers are not close. If the same low-rank transformation parameters are used for processing, it is easy to cause the lower network layer to lose information. Therefore, this step can determine the adjustment coefficient based on the similarity, and use the adjustment coefficient to increase the low-rank transformation parameter of the upper network layer in the adjacent network layer to obtain the low-rank transformation parameter of the lower network layer in the adjacent network layer. After the low-rank transformation parameter is increased, the compression strength received by the lower network layer is reduced, and thus more information can be retained, thereby avoiding the problem of information loss in the lower network layer.

[0132] It is also worth pointing out that from the perspective of the overall model, the low-rank transformation parameters gradually increase as the network layers go deeper, and thus appropriate low-rank transformation parameters can be set for each layer in the deep network. This can avoid the serious problem of information loss in the deep network caused by setting fixed low-rank transformation parameters in related technologies.

[0133] It should be noted that this embodiment does not limit the calculation method of the adjustment coefficient, and it can be set according to actual application requirements, as long as it can ensure that the adjustment coefficient can increase the low-rank transformation parameter of the upper network layer in the adjacent network layer. The similarity can be negatively correlated with the adjustment coefficient, that is, the smaller the similarity, the larger the adjustment coefficient, so that the greater the difference between adjacent network layers, the larger the low-rank transformation parameter obtained by the lower network layer and the more information retained. The following is an introduction to the calculation method of the adjustment coefficient.

[0134] Based on this, determining an adjustment coefficient according to the similarity, and increasing a low-rank transformation parameter of an upper network layer in the adjacent network layer by using the adjustment coefficient may include:

[0135] Step 51: determining a correlation factor according to the similarity; wherein the similarity is positively correlated with the correlation factor;

[0136] Step 52: Taking the reciprocal of the correlation factor as the adjustment coefficient, and using the adjustment coefficient to increase the low-rank transformation parameter of the upper network layer in the adjacent network layer.

[0137] Specifically, the adjustment coefficient can be expressed as:

[0138] ;

[0139] Where R(L(i-1)) represents the low-rank transformation parameter of the (i-1)th layer; Indicates the correlation factor, round() indicates the rounding function, for example, round(1.5)=2; sim i Represents the similarity between the i-th network layer and the i-1-th network layer. It can be seen that the similarity is positively correlated with the correlation factor. The smaller the similarity, the smaller the correlation factor. The correlation factor is negatively correlated with the adjustment coefficient. The smaller the correlation factor, the larger the adjustment coefficient. This ensures that if the similarity is lower, the low-rank conversion parameter of the i-th network layer is larger than that of the (i-1) layer.

[0140] Steps S104 and S105 can be expressed using an adaptive selection function (AF) as follows:

[0141] ;

[0142] Where T represents the preset threshold; F1 is an equivalent function, which can be expressed as: , that is, the low-rank transformation parameters of the i-th layer operator are equal to those of the (i-1)th layer. F2 can be expressed as:

[0143] .

[0144] When the similarity sim i When it is greater than or equal to the threshold, the adaptive selection function AF is F 1 , when the similarity sim i When it is less than the threshold, the adaptive selection function AF is F 2 .

[0145] Based on the above embodiments, the present invention can first obtain a pre-trained language model and preset low-rank conversion parameters, and for the network layers corresponding to the same network layer type in the pre-trained language model, the preset low-rank conversion parameters can be set as the low-rank conversion parameters of the top network layer, and the similarity between adjacent network layers is determined. Subsequently, the present invention determines whether the similarity between adjacent network layers is less than a preset threshold. If the similarity is not less than the preset threshold, it means that the pair of adjacent network layers are relatively similar, and the same low-rank conversion parameters can be used for low-rank conversion, and then the low-rank conversion parameters of the upper network layer in the adjacent network layer can be used as the low-rank conversion parameters of the lower network layer in the adjacent network layer; if the similarity is less than the preset threshold, it means that the pair of adjacent network layers are not similar, and a larger low-rank conversion parameter needs to be set for the lower network layer to retain information, and then the adjustment coefficient can be determined according to the similarity, and the low-rank conversion parameters of the upper network layer in the adjacent network layer are increased by using the adjustment coefficient to obtain the low-rank conversion parameters of the lower network layer in the adjacent network layer. In this way, the present invention can adaptively set the low-rank conversion parameters for the lower network layer according to the similarity between adjacent network layers, and generally ensure that the deeper the network layer, the larger the low-rank conversion parameter. This can effectively ensure the low-rank conversion effect of the model and reduce the problem of information loss in deep networks, thereby improving the efficiency and performance of model fine-tuning training.

[0146] Based on the above embodiment, the purpose of the low-rank conversion parameter and the fine-tuning process of the pre-trained language model are introduced below. In one possible case, the method may also include:

[0147] S201. Perform matrix decomposition on the weight matrix of the network layer according to the low-rank transformation parameters to obtain a low-rank transformation weight matrix corresponding to the network layer.

[0148] For easier understanding, please refer to Figure 2 , Figure 2 A schematic diagram of a matrix decomposition provided by an embodiment of the present invention, x represents the input matrix, h represents the output matrix, W represents the weight matrix before decomposition, A and B represent the weight matrix after decomposition, and the A matrix is ​​initialized using a random Gaussian distribution (A=N(0,δ 2 )), the B matrix is ​​initialized with all 0s (B=0). In this embodiment, the weight matrix W of the network layer can be d×k Decomposed into two matrices A d×r and B r×k, the two are connected by a low-rank transformation parameter r, where the rank r is much smaller than d and k. The amount of training parameters is reduced from the original d*k to (d*r+r*k), which greatly reduces the amount of training parameters. At the same time, the gradient values ​​that need to be saved are much less, thus saving a lot of video memory. At the same time, since different network layers correspond to different low-rank transformation parameters, on the whole, the deeper the network layer, the larger the corresponding low-rank transformation parameter. In this way, on the basis of reducing the amount of parameters, the deeper the network layer, the more information it retains, thereby avoiding the loss of key information.

[0149] S202: Fine-tune the low-rank transformation weight matrix of the network layer.

[0150] In this step, the original weight matrix of the network layer can be frozen, and only the low-rank transformation weight matrix can be fine-tuned, so as to achieve the effect of efficient fine-tuning of the model. The specific process of fine-tuning training will be introduced below.

[0151] Based on this, fine-tuning the low-rank transformation weight matrix of the network layer may include:

[0152] S301, obtaining a preset training text.

[0153] In this step, the preset training text may come from a knowledge base in a specific field. This embodiment does not limit the specific content and quantity of the preset training text, which may be set according to actual application requirements.

[0154] S302. Process the preset training text layer by layer using each network layer in the pre-trained language model to obtain model output data; wherein the output of the network layer is fused output data obtained by fusing the first output data and the second output data, the first output data is the processing result of the preset training text by the weight matrix of the network layer, and the second output data is the processing result of the preset training text by the low-rank transformation weight matrix of the network layer.

[0155] In this step, the preset training text can be input into the pre-trained language model, and each network layer in the pre-trained language model is used to process the preset training text layer by layer to obtain model output data. It should be pointed out that in each network layer, the features corresponding to the preset training text need to be processed by the original weight matrix and the low-rank conversion weight matrix of the network layer respectively, and the first output data obtained by the weight matrix processing and the second output data obtained by the low-rank conversion weight matrix processing are obtained, and then the first output data and the second output data are fused to obtain the fused output data output by the network layer.

[0156] The following is an introduction to the processing of the preset training text in each network layer. Based on this, the preset training text is processed layer by layer using each network layer in the pre-trained language model to obtain model output data, which may include:

[0157] Step 51: Use the first network layer in the pre-trained language model as the current network layer, use the preset training text as the data to be processed, and input the data to be processed into the current network layer.

[0158] Step 52: Use the weight matrix of the current network layer to process the data to be processed to obtain the first output data.

[0159] Step 53: Use the low-rank transformation weight matrix of the current network layer to process the data to be processed to obtain the second output data, and use the scaling parameter to scale the second output data.

[0160] Step 54: superimpose the first output data with the second output data after scaling processing to obtain the fused output data.

[0161] In steps 53 and 54, after obtaining the second output data, the second output data can be scaled using the scaling parameter, and the first output data and the scaled second output data are superimposed, which is equivalent to weighted superposition of the first output data and the second output data to obtain fused output data.

[0162] It should be noted that the present embodiment does not limit the specific value of the scaling parameter, and it can be set according to actual application requirements. In addition, the scaling parameter can also be a value adaptively set for each network layer. For example, the low-rank transformation parameter of each network layer can be obtained, and the scaling parameter corresponding to the network layer can be determined using the low-rank transformation parameter and the preset scaling factor.

[0163] Based on this, before scaling the second output data using the scaling parameter, the method further includes:

[0164] Step 61: Determine a scaling parameter corresponding to the current network layer by using the low-rank transformation parameter of the current network layer and a preset scaling factor;

[0165] The scaling process of the second output data by using the scaling parameter includes:

[0166] Step 62: Scale the second output data using the scaling parameter corresponding to the current network layer.

[0167] It can be seen that the present application can adaptively generate scaling parameters for each network layer according to the low-rank transformation parameters of each network layer, so as to ensure that the superposition processing of the first output data and the second output data is also related to the network layer. The superposition processing of the first output data and the second output data can be expressed as:

[0168] ;

[0169] Among them, h represents the final output result, x represents the input data, W represents the original weight matrix, and Wx represents the first output data. represents the second output data, k represents the scaling factor, B and A represent the dot product of the low-rank transformation weight matrix, α represents the scaling parameter, and r represents the low-rank transformation parameter.

[0170] Step 55: Determine whether there is a lower network layer that has not processed the preset training text.

[0171] Step 56: If yes, the fused output data of the current network layer is used as the data to be processed, the current network layer is updated to the lower network layer of the current network layer, and the step of inputting the data to be processed into the current network layer is entered.

[0172] Step 57: If not, the fused output data of the current network layer is used as the model output data.

[0173] S303: determining a loss value based on the model output data, and performing parameter update on the low-rank transformation weight matrix of each network layer according to the loss value.

[0174] In this step, the loss value can be determined based on the model output data, and the parameters of the low-rank transformation weight matrix of each network layer can be updated according to the loss value, so that the parameter value of the low-rank transformation weight matrix can be used to save the information added by the preset training text. Furthermore, in actual reasoning, the general knowledge saved by the original weight matrix of the network layer and the fine-tuning knowledge saved by the low-rank transformation weight matrix of the network layer can be used together for text parsing and generation, thereby ensuring model performance.

[0175] It should be noted that this embodiment does not limit the specific process of determining the loss value, and reference may be made to related technologies of artificial intelligence and language models.

[0176] Based on the above embodiment, in order to effectively evaluate the impact of different preset low-rank conversion parameters on model performance, this embodiment can also preset multiple groups of different preset low-rank conversion parameter groups, determine the low-rank conversion parameters for the network layer in the pre-trained language model based on each preset low-rank conversion parameter group, and fine-tune the pre-trained language model to determine the model performance values ​​corresponding to different preset low-rank conversion parameter groups, and then select the best preset low-rank conversion parameter group. The selection process of the preset low-rank conversion parameter group is introduced below. In one possible case, the method may also include:

[0177] S401. Create different preset low-rank transformation parameter groups; wherein the preset low-rank transformation parameter groups store preset low-rank transformation parameters corresponding to each type of network layer, and different preset low-rank transformation parameter groups store different preset low-rank transformation parameters.

[0178] S402: Determine different low-rank transformation parameters for the network layer in the pre-trained language model based on the different preset low-rank transformation parameter groups.

[0179] S403: Fine-tune each network layer in the pre-trained language model based on different low-rank transformation parameters to determine model performance values ​​corresponding to different preset low-rank transformation parameter groups.

[0180] S404: Determine an optimal preset low-rank transformation parameter group according to the model performance value.

[0181] In this embodiment, different preset low-rank conversion parameter groups can be used to fine-tune multiple different model versions, and the best preset low-rank conversion parameter group can be screened out according to the model performance value of each model version to improve the model fine-tuning performance. It is worth noting that in this embodiment, only the preset low-rank conversion parameters need to be determined for the top network layer, which can effectively reduce the difficulty of manual debugging of the preset low-rank conversion parameters.

[0182] It should be noted that this embodiment does not limit how to determine the model performance value. For example, the model performance value can be determined using indicators such as accuracy, precision, recall, F1 score, AUC-ROC and cross entropy loss.

[0183] Based on the above embodiment, the process of generating text based on user input text using the pre-trained language model after fine-tuning training is introduced below. Figure 3 , Figure 3 A flowchart of a text generation method provided by an embodiment of the present invention. In a possible case, the method may also include:

[0184] S501. Obtain user input text and a pre-trained language model; wherein the network layer in the pre-trained language model includes a weight matrix and a low-rank transformation weight matrix, and the low-rank transformation weight matrix is ​​generated using the weight matrix and the low-rank transformation parameters corresponding to the network layer, and the low-rank transformation parameters are generated according to the parameter setting method as described above.

[0185] In this step, the user input text and the pre-trained language model can be obtained. The user input text contains the user's intention. The pre-trained language model has been fine-tuned and trained, and each network layer contains the original weight matrix and the low-rank conversion weight matrix. The weight parameters of the weight matrix are used to save general knowledge, and the weight parameters of the low-rank conversion weight matrix are used to save the specified domain knowledge. In this way, when generating text, the general knowledge saved by the original weight matrix of the network layer and the specified domain knowledge saved by the low-rank conversion weight matrix of the network layer can be used together for text parsing and generation, thereby ensuring the performance of the model.

[0186] S502. Process the user input text layer by layer using each network layer in the pre-trained language model to obtain model output data; wherein the output of the network layer is fused output data obtained by fusing the first output data and the second output data, the first output data is the result of processing the user input text by the weight matrix of the network layer, and the second output data is the result of processing the user input text by the low-rank transformation weight matrix of the network layer.

[0187] In this step, similar to fine-tuning training, when each network layer processes the user input text, the weight matrix and the low-rank transformation weight matrix in the network layer must first be processed separately to obtain the first output data obtained by the weight matrix processing and the second output data obtained by the low-rank transformation weight matrix processing, and then the first output data and the second output data are fused to obtain the fused output data output by the network layer.

[0188] The following is an introduction to the processing of user input text in each network layer. Based on this, the user input text is processed layer by layer using each network layer in the pre-trained language model to obtain model output data, which may include:

[0189] Step 61: Use the first network layer in the pre-trained language model as the current network layer, use the user input text as the data to be processed, and input the data to be processed into the current network layer.

[0190] Step 62: Use the weight matrix of the current network layer to process the data to be processed to obtain the first output data.

[0191] Step 63: Use the low-rank transformation weight matrix of the current network layer to process the data to be processed to obtain the second output data, and use the scaling parameter to scale the second output data.

[0192] Step 64: superimpose the first output data with the scaled second output data to obtain the fused output data.

[0193] In steps 63 and 64, after obtaining the second output data, the second output data can be scaled using the scaling parameter, and the first output data and the scaled second output data are superimposed, which is equivalent to weighted superposition of the first output data and the second output data to obtain fused output data.

[0194] Furthermore, the scaling parameter may be a value adaptively set for each network layer. For example, a low-rank transformation parameter of each network layer may be obtained, and the scaling parameter corresponding to the network layer may be determined using the low-rank transformation parameter and a preset scaling factor.

[0195] Based on this, before scaling the second output data using the scaling parameter, the method may further include:

[0196] Step 71: Determine a scaling parameter corresponding to the current network layer by using the low-rank transformation parameter of the current network layer and a preset scaling factor;

[0197] The scaling process of the second output data by using the scaling parameter includes:

[0198] Step 72: Scale the second output data using the scaling parameter corresponding to the current network layer.

[0199] It can be seen that the present application can adaptively generate scaling parameters for each network layer according to the low-rank transformation parameters of each network layer, so as to ensure that the superposition processing of the first output data and the second output data is also related to the network level.

[0200] Step 65: Determine whether there is a lower network layer that has not processed the user input text;

[0201] Step 66: If yes, the fused output data of the current network layer is used as the data to be processed, the current network layer is updated to the lower network layer of the current network layer, and the step of inputting the data to be processed into the current network layer is entered;

[0202] Step 67: If not, the fused output data of the current network layer is used as the model output data.

[0203] S503: Generate text based on the model output data.

[0204] It should be noted that this embodiment does not limit how to generate text based on the model output data. For example, text sampling can be performed based on the model output data to select each word in the generated text one by one. Please refer to the specific content of the language model.

[0205] Based on the above embodiment, the above parameter setting method is introduced below based on specific examples and schematic diagrams. This process may include:

[0206] (1) Load the pre-trained model and fine-tune the dataset.

[0207] (2) Predefined low-rank approximation module. The linear layer in the feedforward network in the pre-trained model is defined as an adaptive low-rank approximation module, assuming that N has 24 layers in total.

[0208] (3) Adaptive low-rank approximation fine-tuning module. In this method, for each category, all modules first calculate their respective low-rank transformation parameters and convert the pre-trained weights W∈ Decomposed into the dot product of two matrices BA, where B∈ , A∈ . Define the similarity threshold T=0.5. Please refer to Figure 4 , Figure 4 This is a schematic diagram of a parameter setting process provided by an embodiment of the present invention, where i is the serial number of the linear layer. The specific operation process is as follows:

[0209] A. For the first linear layer Linear, set its operator W 1 The low-rank transformation parameter r 1 =2.

[0210] B. Calculate the second linear layer operator W 2 With W 1 The similarity value, sim 1 =0.3.

[0211] C. Due to sim 1 is less than the threshold T(0.5), then the adaptive selection function AF=F 2 , so r 2 =2 / 2 (-2) =8.

[0212] D. Use the same method to calculate the low-rank transformation parameters of 24 linear layers in sequence, recorded as r = [2, 8, 16, 16, 32, ..., 128, 128, 256, 512]. It can be seen that the low-rank transformation parameters of the deep network are getting larger and larger, indicating that it is not suitable to use low-rank approximation and more complete information needs to be retained.

[0213] (4) On the code dataset, fine-tune the pre-trained model based on 24 low-rank transformation parameters and save it.

[0214] (5) Model application. In the actual reasoning process, for the input data x, the output of the original weight and the output after passing through the A and B matrices are superimposed as follows to obtain the final output result:

[0215] ;

[0216] Among them, h represents the final output result, W represents the original weight matrix, represents the low-rank transformation weight matrix, k represents the scaling factor, α represents the scaling parameter, and r represents the low-rank transformation parameter.

[0217] The parameter setting device, text generation method, electronic device, computer program product, and non-volatile computer-readable storage medium provided in the embodiments of the present invention are introduced below. The parameter setting device, text generation method, electronic device, computer program product, and non-volatile computer-readable storage medium described below can be referenced to the parameter setting method and text generation method described above.

[0218] Please refer to Figure 5 , Figure 5 A structural block diagram of a parameter setting device provided by an embodiment of the present invention, the device may include:

[0219] An acquisition module 501 is used to acquire a pre-trained language model and preset low-rank conversion parameters;

[0220] A network layer processing module 502 is used to set the preset low-rank transformation parameter as the low-rank transformation parameter of the top network layer for the network layers corresponding to the same network layer type in the pre-trained language model, and determine the similarity between adjacent network layers;

[0221] A first low-rank conversion parameter setting module 503 is configured to use the low-rank conversion parameters of the upper network layer in the adjacent network layers as the low-rank conversion parameters of the lower network layer in the adjacent network layers if the similarity is not less than a preset threshold;

[0222] The second low-rank conversion parameter setting module 504 is used to determine an adjustment coefficient according to the similarity if the similarity is less than the preset threshold, and use the adjustment coefficient to increase the low-rank conversion parameter of the upper network layer in the adjacent network layer to obtain the low-rank conversion parameter of the lower network layer in the adjacent network layer.

[0223] Optionally, the second low-rank transformation parameter setting module 504 may include:

[0224] A correlation factor determination submodule, used to determine the correlation factor according to the similarity; wherein the similarity is positively correlated with the correlation factor;

[0225] The adjustment submodule is used to use the reciprocal of the correlation factor as the adjustment coefficient, and use the adjustment coefficient to increase the low-rank conversion parameter of the upper network layer in the adjacent network layer.

[0226] Optionally, the acquisition module 501 may include:

[0227] A classification submodule, used to determine the type of network layer corresponding to each network layer in the pre-trained language model;

[0228] The acquisition submodule is used to obtain the preset low-rank transformation parameters corresponding to each type of network layer.

[0229] Optionally, the network layer processing module 502 may include:

[0230] A parameter setting submodule is used to set the preset low-rank conversion parameters corresponding to the network layer type as the low-rank conversion parameters of the top network layer corresponding to the network layer type in the pre-trained language model.

[0231] Optionally, the device may further include:

[0232] A creation module, used to create different preset low-rank transformation parameter groups; wherein the preset low-rank transformation parameter groups store the preset low-rank transformation parameters corresponding to each of the network layer types, and different preset low-rank transformation parameter groups store different preset low-rank transformation parameters;

[0233] A parameter setting module, used to determine different low-rank transformation parameters for the network layer in the pre-trained language model based on the different preset low-rank transformation parameter groups;

[0234] A multi-version fine-tuning module, used to perform fine-tuning training on each network layer in the pre-trained language model based on different low-rank transformation parameters, and determine model performance values ​​corresponding to different preset low-rank transformation parameter groups;

[0235] A selection module is used to determine the best preset low-rank transformation parameter group according to the model performance value.

[0236] Optionally, the network layer processing module 502 may include:

[0237] The similarity calculation submodule is used to determine the characteristic distance between the weight matrices of the adjacent network layers and use the characteristic distance as the similarity.

[0238] Optionally, the similarity calculation submodule may include:

[0239] A first decomposition unit is used to perform singular value decomposition on a first weight matrix of an upper network layer in an adjacent network layer to obtain a first non-negative diagonal matrix and a first right singular matrix, and determine the rank of the first non-negative diagonal matrix;

[0240] A first screening unit is used to determine a first screening quantity according to the rank of the first non-negative diagonal matrix, and to screen the first right singular matrix for the first singular vectors whose number is the first screening quantity;

[0241] A second decomposition unit is used to perform singular value decomposition on a second weight matrix corresponding to a lower network layer in an adjacent network layer to obtain a second non-negative diagonal matrix and a second right singular matrix, and determine the rank of the second non-negative diagonal matrix;

[0242] A second screening unit is used to determine a second screening quantity according to the rank of the second non-negative diagonal matrix, and screen the front second singular vectors whose number is the second screening quantity in the second right singular matrix;

[0243] A computing unit is used to determine the similarity using the first singular vector, the rank of the first non-negative diagonal matrix, the second singular vector, and the rank of the second non-negative diagonal matrix.

[0244] Optionally, the device may further include:

[0245] A low-rank conversion weight matrix generation module is used to perform matrix decomposition on the weight matrix of the network layer according to the low-rank conversion parameters to obtain a low-rank conversion weight matrix corresponding to the network layer;

[0246] The fine-tuning module is used to perform fine-tuning training on the low-rank transformation weight matrix of the network layer.

[0247] Optionally, the fine-tuning module may include:

[0248] The acquisition submodule is used to obtain the preset training text;

[0249] A processing submodule, used to process the preset training text layer by layer using each network layer in the pre-trained language model to obtain model output data; wherein the output of the network layer is fused output data obtained by fusion processing of the first output data and the second output data, the first output data is the processing result of the preset training text by the weight matrix of the network layer, and the second output data is the processing result of the preset training text by the low-rank transformation weight matrix of the network layer;

[0250] A parameter updating module is used to determine a loss value based on the model output data, and to update the parameters of the low-rank transformation weight matrix of each network layer according to the loss value.

[0251] Optionally, the processing submodule may include:

[0252] A setting unit, used to use the first network layer in the pre-trained language model as the current network layer, use the preset training text as the data to be processed, and input the data to be processed into the current network layer;

[0253] A first processing unit, configured to process the data to be processed using the weight matrix of the current network layer to obtain the first output data;

[0254] A second processing unit, configured to process the data to be processed by using the low-rank transformation weight matrix of the current network layer to obtain the second output data, and perform scaling processing on the second output data by using a scaling parameter;

[0255] a fusion unit, configured to superimpose the first output data with the second output data after scaling processing to obtain the fused output data;

[0256] A judging unit, used for judging whether there is a lower network layer that has not processed the preset training text;

[0257] A loop control unit, configured to, if available, use the fused output data of the current network layer as the data to be processed, update the current network layer to a lower network layer of the current network layer, and enter a step of inputting the data to be processed into the current network layer;

[0258] The output unit is used to use the fused output data of the current network layer as the model output data if it is not available.

[0259] Optionally, the device may further include:

[0260] A scaling parameter determination module, used to determine the scaling parameter corresponding to the current network layer by using the low-rank transformation parameter of the current network layer and a preset scaling factor;

[0261] The second processing unit may include:

[0262] The scaling subunit is used to scale the second output data using the scaling parameters corresponding to the current network layer.

[0263] Please refer to Figure 6 , Figure 6 A structural block diagram of a text generation device provided by an embodiment of the present invention, the device may include:

[0264] An acquisition module 601 is used to acquire user input text and a pre-trained language model; wherein the network layer in the pre-trained language model comprises a weight matrix and a low-rank conversion weight matrix, the low-rank conversion weight matrix is ​​generated using the weight matrix and the low-rank conversion parameters corresponding to the network layer, and the low-rank conversion parameters are generated according to the parameter setting method as described above;

[0265] The model processing module 602 is used to process the user input text layer by layer using each network layer in the pre-trained language model to obtain model output data; wherein the output of each network layer is fused output data obtained by fusion processing of the first output data and the second output data, the first output data is the processing result of the weight matrix of the network layer on the user input text, and the second output data is the processing result of the low-rank transformation weight matrix of the network layer on the user input text;

[0266] The text generation module 603 is used to generate text based on the model output data.

[0267] Optionally, the model processing module 602 may include:

[0268] Setting a submodule, used to use the first network layer in the pre-trained language model as the current network layer, use the user input text as the data to be processed, and input the data to be processed into the current network layer;

[0269] A first processing submodule, configured to process the data to be processed using the weight matrix of the current network layer to obtain the first output data;

[0270] A second processing submodule, configured to process the data to be processed using the low-rank transformation weight matrix of the current network layer to obtain the second output data, and perform scaling processing on the second output data using a scaling parameter;

[0271] a fusion submodule, configured to superimpose the first output data with the second output data after scaling processing to obtain the fused output data;

[0272] A judgment submodule, used for judging whether there is a lower network layer that has not processed the user input text;

[0273] A loop control submodule, for, if available, using the fused output data of the current network layer as the data to be processed, updating the current network layer to a lower network layer of the current network layer, and entering a step of inputting the data to be processed into the current network layer;

[0274] The output submodule is used to use the fused output data of the current network layer as the model output data if it is not available.

[0275] Optionally, the device may further include:

[0276] A scaling parameter determination module, used to determine the scaling parameter corresponding to the current network layer by using the low-rank transformation parameter of the current network layer and a preset scaling factor;

[0277] The second processing submodule may include:

[0278] A scaling unit is used to scale the second output data using the scaling parameter corresponding to the current network layer.

[0279] Please refer to Figure 7 , Figure 7 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The embodiment of the present invention provides an electronic device 10, including a processor 11 and a memory 12; wherein the memory 12 is used to store a computer program; and the processor 11 is used to execute the parameter setting method and the text generation method provided in the aforementioned embodiment when executing the computer program.

[0280] For the specific process of the above-mentioned parameter setting method and text generation method, please refer to the corresponding content provided in the aforementioned embodiments, which will not be repeated here.

[0281] Furthermore, the memory 12 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0282] In addition, the electronic device 10 also includes a power supply 13, a communication interface 14, an input / output interface 15 and a communication bus 16; wherein the power supply 13 is used to provide working voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input / output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0283] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the parameter setting method and text generation method described in the above embodiment.

[0284] Since the embodiments of the computer program product part correspond to the embodiments of the parameter setting method and text generation method part, please refer to the description of the embodiments of the parameter setting method and text generation method part for the embodiments of the computer program product part, and will not be repeated here.

[0285] An embodiment of the present invention further provides a non-volatile computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the parameter setting method and the text generation method described in the above embodiment are implemented.

[0286] Since the embodiments of the non-volatile computer-readable storage medium part correspond to the embodiments of the parameter setting method and text generation method part, the embodiments of this part please refer to the description of the embodiments of the parameter setting method and text generation method part, and will not be repeated here.

[0287] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0288] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0289] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0290] The parameter setting method, text generation method, device, equipment, and program product provided by the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A parameter setting method, characterized in that: include: Get the pre-trained language model and preset low-rank transformation parameters; For network layers corresponding to the same network layer type in the pre-trained language model, setting the preset low-rank transformation parameters as low-rank transformation parameters of the top network layer, and determining the similarity between adjacent network layers; If the similarity is not less than a preset threshold, using the low-rank transformation parameters of the upper network layer in the adjacent network layers as the low-rank transformation parameters of the lower network layer in the adjacent network layers; If the similarity is less than the preset threshold, an adjustment coefficient is determined according to the similarity, and the low-rank transformation parameter of the upper network layer in the adjacent network layer is increased by using the adjustment coefficient to obtain the low-rank transformation parameter of the lower network layer in the adjacent network layer.

2. The parameter setting method according to claim 1, characterized in that: Determining an adjustment coefficient according to the similarity, and increasing a low-rank transformation parameter of an upper network layer in the adjacent network layer by using the adjustment coefficient, including: Determining a correlation factor according to the similarity; wherein the similarity is positively correlated with the correlation factor; The reciprocal of the correlation factor is used as the adjustment coefficient, and the low-rank transformation parameter of the upper network layer in the adjacent network layer is increased by using the adjustment coefficient.

3. The parameter setting method according to claim 1, characterized in that: Get the preset low-rank transformation parameters, including: Determine the network layer type corresponding to each network layer in the pre-trained language model; Obtain preset low-rank transformation parameters corresponding to each of the network layer types.

4. The parameter setting method according to claim 3, characterized in that: For the network layers corresponding to the same network layer type in the pre-trained language model, setting the preset low-rank conversion parameter as the low-rank conversion parameter of the top network layer includes: For network layers corresponding to the same network layer type in the pre-trained language model, the preset low-rank transformation parameters corresponding to the network layer type are set as low-rank transformation parameters of the top network layer corresponding to the network layer type.

5. The parameter setting method according to claim 3, characterized in that: Also includes: Creating different preset low-rank transformation parameter groups; wherein the preset low-rank transformation parameter groups store the preset low-rank transformation parameters corresponding to each of the network layer types, and different preset low-rank transformation parameter groups store different preset low-rank transformation parameters; Determining different low-rank transformation parameters for the network layer in the pre-trained language model based on different preset low-rank transformation parameter groups; Fine-tune each network layer in the pre-trained language model based on different low-rank transformation parameters to determine model performance values ​​corresponding to different preset low-rank transformation parameter groups; An optimal preset low-rank transformation parameter set is determined based on the model performance value.

6. The parameter setting method according to claim 1, characterized in that: Determine the similarity between adjacent network layers, including: Determine the characteristic distance between the weight matrices of the adjacent network layers, and use the characteristic distance as the similarity.

7. The parameter setting method according to claim 6, characterized in that: Determining the characteristic distance between the weight matrices of the adjacent network layers includes: Performing singular value decomposition on a first weight matrix of an upper network layer in an adjacent network layer to obtain a first non-negative diagonal matrix and a first right singular matrix, and determining the rank of the first non-negative diagonal matrix; Determine a first screening quantity according to the rank of the first non-negative diagonal matrix, and screen the first right singular matrix for the first singular vectors whose number is the first screening quantity; Performing singular value decomposition on a second weight matrix corresponding to a lower network layer in an adjacent network layer to obtain a second non-negative diagonal matrix and a second right singular matrix, and determining the rank of the second non-negative diagonal matrix; Determine a second screening number according to the rank of the second non-negative diagonal matrix, and screen the first second singular vectors whose number is the second screening number in the second right singular matrix; The similarity is determined using the first singular vectors, the rank of the first non-negative diagonal matrix, the second singular vectors, and the rank of the second non-negative diagonal matrix.

8. The parameter setting method according to claim 1, characterized in that: The pre-trained language model is composed of multiple transformer network layers connected in series.

9. The parameter setting method according to any one of claims 1 to 8, characterized in that: Also includes: Performing matrix decomposition on the weight matrix of the network layer according to the low-rank transformation parameter to obtain a low-rank transformation weight matrix corresponding to the network layer; The low-rank transformation weight matrix of the network layer is fine-tuned and trained.

10. The parameter setting method according to claim 9, characterized in that: Fine-tuning the low-rank transformation weight matrix of the network layer includes: Get the preset training text; The preset training text is processed layer by layer using each network layer in the pre-trained language model to obtain model output data; wherein the output of the network layer is fused output data obtained by fusion processing of the first output data and the second output data, the first output data is the processing result of the preset training text by the weight matrix of the network layer, and the second output data is the processing result of the preset training text by the low-rank transformation weight matrix of the network layer; A loss value is determined based on the model output data, and parameters of the low-rank transformation weight matrix of each network layer are updated according to the loss value.

11. The parameter setting method according to claim 10, characterized in that: The preset training text is processed layer by layer using each network layer in the pre-trained language model to obtain model output data, including: Using the first network layer in the pre-trained language model as the current network layer, using the preset training text as the data to be processed, and inputting the data to be processed into the current network layer; Processing the data to be processed using the weight matrix of the current network layer to obtain the first output data; Using the low-rank transformation weight matrix of the current network layer to process the data to be processed to obtain the second output data, and using the scaling parameter to scale the second output data; Superimposing the first output data with the second output data after scaling processing to obtain the fused output data; Determining whether there is a lower network layer that has not processed the preset training text; If yes, the fused output data of the current network layer is used as the data to be processed, the current network layer is updated to the lower network layer of the current network layer, and the step of inputting the data to be processed into the current network layer is entered; If not, the fused output data of the current network layer is used as the model output data.

12. The parameter setting method according to claim 11, characterized in that: Before scaling the second output data using the scaling parameter, the method further includes: Determine a scaling parameter corresponding to the current network layer by using a low-rank transformation parameter of the current network layer and a preset scaling factor; Scaling the second output data using a scaling parameter includes: The second output data is scaled using a scaling parameter corresponding to the current network layer.

13. A text generation method, characterized in that: include: Obtaining user input text and a pre-trained language model; wherein the network layer in the pre-trained language model comprises a weight matrix and a low-rank conversion weight matrix, the low-rank conversion weight matrix is ​​generated using the weight matrix and the low-rank conversion parameters corresponding to the network layer, and the low-rank conversion parameters are generated according to the parameter setting method according to any one of claims 1 to 12; Using each network layer in the pre-trained language model to process the user input text layer by layer to obtain model output data; wherein the output of the network layer is fused output data obtained by fusion processing of the first output data and the second output data, the first output data is the result of the weight matrix of the network layer processing the user input text, and the second output data is the result of the low-rank transformation weight matrix of the network layer processing the user input text; Text generation is performed based on the model output data.

14. The text generation method according to claim 13, characterized in that: The user input text is processed layer by layer using each network layer in the pre-trained language model to obtain model output data, including: Using the first network layer in the pre-trained language model as the current network layer, using the user input text as data to be processed, and inputting the data to be processed into the current network layer; Processing the data to be processed using the weight matrix of the current network layer to obtain the first output data; Using the low-rank transformation weight matrix of the current network layer to process the data to be processed to obtain the second output data, and using the scaling parameter to scale the second output data; Superimposing the first output data with the second output data after scaling processing to obtain the fused output data; Determining whether there is a lower network layer that has not processed the user input text; If yes, the fused output data of the current network layer is used as the data to be processed, the current network layer is updated to the lower network layer of the current network layer, and the step of inputting the data to be processed into the current network layer is entered; If not, the fused output data of the current network layer is used as the model output data.

15. The text generation method according to claim 14, characterized in that: Before scaling the second output data using the scaling parameter, the method further includes: Determine a scaling parameter corresponding to the current network layer by using a low-rank transformation parameter of the current network layer and a preset scaling factor; Scaling the second output data using a scaling parameter includes: The second output data is scaled using a scaling parameter corresponding to the current network layer.

16. A parameter setting device, characterized in that: include: An acquisition module is used to obtain a pre-trained language model and preset low-rank conversion parameters; A network layer processing module, for setting the preset low-rank transformation parameters as low-rank transformation parameters of the top network layer for the network layers corresponding to the same network layer type in the pre-trained language model, and determining the similarity between adjacent network layers; A first low-rank conversion parameter setting module, configured to use the low-rank conversion parameters of the upper network layer in the adjacent network layers as the low-rank conversion parameters of the lower network layer in the adjacent network layers if the similarity is not less than a preset threshold; The second low-rank conversion parameter setting module is used to determine the adjustment coefficient according to the similarity if the similarity is less than the preset threshold, and use the adjustment coefficient to increase the low-rank conversion parameter of the upper network layer in the adjacent network layer to obtain the low-rank conversion parameter of the lower network layer in the adjacent network layer.

17. A text generation device, characterized in that: include: An acquisition module, used to acquire user input text and a pre-trained language model; wherein the network layer in the pre-trained language model comprises a weight matrix and a low-rank conversion weight matrix, wherein the low-rank conversion weight matrix is ​​generated using the weight matrix and the low-rank conversion parameters corresponding to the network layer, and the low-rank conversion parameters are generated according to the parameter setting method according to any one of claims 1 to 12; A model processing module, used to process the user input text layer by layer using each network layer in the pre-trained language model to obtain model output data; wherein the output of each network layer is fused output data obtained by fusion processing of first output data and second output data, the first output data is the processing result of the user input text by the weight matrix of the network layer, and the second output data is the processing result of the user input text by the low-rank transformation weight matrix of the network layer; A text generation module is used to generate text based on the model output data.

18. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the parameter setting method as described in any one of claims 1 to 12 or the text generation method as described in any one of claims 13 to 15 when executing the computer program.

19. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the parameter setting method according to any one of claims 1 to 12 or the text generation method according to any one of claims 13 to 15 is implemented.

20. A non-volatile computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the parameter setting method according to any one of claims 1 to 12 or the text generation method according to any one of claims 13 to 15 is implemented.

Images