Parameter selection method and device, text generation method and device, equipment and medium

Abstract

The invention provides a parameter selection method and device, electronic equipment and a storage medium, and relates to the field of artificial intelligence, and the method comprises the steps: obtaining a reference question and answer text pair and a pre-training language model, and extracting a weight parameter from the pre-training language model; carrying out correlation analysis on the weight parameters to obtain a correlation relationship among the weight parameters, and dividing the weight parameters into a plurality of correlation parameter groups according to the correlation relationship; for each associated parameter group, performing multiple times of numerical adjustment on the associated parameter group in the pre-training language model, and determining a performance index value obtained by the pre-training language model in the preset performance indexes by using the reference question and answer text pair after each time of numerical adjustment is completed; determining a sensitive parameter group and a non-sensitive parameter group in the associated parameter group according to the change condition of the performance index value corresponding to the associated parameter group in each numerical value adjustment, so as to determine the weight parameter in the sensitive parameter group as a to-be-trained weight parameter; the to-be-trained weight parameter selection efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a parameter selection method, a text generation method, a device, equipment and a medium. Background Art

[0002] A pre-trained language model is a machine learning model that has been trained with a large amount of text data, and is mainly used to parse and generate natural language text. The common application scenario of this model is: receiving question text input by the user, and outputting output text that meets the user's needs based on the question text. In the related art, the pre-trained language model needs to undergo fine-tuning training to meet the needs of specific application scenarios, and the fine-tuning training requires relevant personnel to decide which part of the weight parameters in the model to update. However, the related art lacks the corresponding means to intuitively and comprehensively analyze the weights of the deep learning model, which is not convenient for understanding the impact of the weight parameters on the model performance, and is likely to affect the choice of relevant personnel to update the weight parameters. For example, it is difficult for relevant personnel to fine-tune the appropriate weight parameters so that the trained model can better output output text that meets user needs. Summary of the invention

[0003] The purpose of the present invention is to provide a parameter selection method, a text generation method, an apparatus, a device and a medium, which can automatically analyze the correlation between each weight parameter in a pre-trained language model and its contribution to model reasoning, thereby improving the efficiency of relevant personnel in further training the pre-trained language model.

[0004] In order to solve the above technical problems, the present invention provides a parameter selection method, comprising:

[0005] Obtaining a benchmark question-answer text pair and a pre-trained language model, and extracting weight parameters from the pre-trained language model;

[0006] Performing correlation analysis on the weight parameters to obtain correlation relationships between the weight parameters, and dividing the weight parameters into a plurality of correlation parameter groups according to the correlation relationships;

[0007] For each associated parameter group, multiple numerical adjustments are made to the associated parameter group in the pre-trained language model, and when each numerical adjustment is completed, the benchmark question-answer text pair is used to determine the performance indicator value obtained by the pre-trained language model in a preset performance indicator;

[0008] According to the change of the performance indicator value corresponding to the associated parameter group in each numerical adjustment, a sensitive parameter group and a non-sensitive parameter group are determined in the associated parameter group, so as to determine the weight parameters in the sensitive parameter group as the weight parameters to be trained.

[0009] Optionally, performing correlation analysis on the weight parameters to obtain correlation relationships between the weight parameters, and dividing the weight parameters into a plurality of correlation parameter groups according to the correlation relationships, includes:

[0010] Calculating the correlation coefficient between each of the weight parameters;

[0011] Classify a pair of weight parameters whose correlation coefficient is greater than a preset threshold into an associated parameter group;

[0012] Or, clustering all the weight parameters according to the values ​​of the weight parameters to obtain a plurality of parameter clustering categories;

[0013] Weight parameters that belong to the same parameter cluster category are grouped into an associated parameter group.

[0014] Optionally, performing multiple numerical adjustments on the associated parameter group in the pre-trained language model includes:

[0015] The associated parameter group is adjusted multiple times in the pre-trained language model according to a preset numerical adjustment strategy; the preset numerical adjustment strategy includes any one or more of a numerical adjustment range, a numerical adjustment step or a random adjustment strategy.

[0016] Optionally, after determining the sensitive parameter group and the non-sensitive parameter group in the associated parameter group, the method further includes:

[0017] Using a variety of chart generation methods to convert the correlation between the weight parameters, the change of the performance indicator value of the sensitive parameter group, and the change of the performance indicator value of the non-sensitive parameter group into a variety of visual charts;

[0018] The visual diagram is output to a display device.

[0019] Optionally, the reference question-answer text pair includes a question text and a reference answer text;

[0020] The step of using the question-answer text pair to determine the performance indicator value obtained by the pre-trained language model in the preset performance indicator when each value adjustment is completed includes:

[0021] When each value adjustment is completed, the question text is input into the pre-trained language model to obtain a generated answer text;

[0022] Calculating text similarity between the generated answer text and the reference answer text;

[0023] Determine the loss value of the pre-trained language model generated in this numerical adjustment according to the text similarity;

[0024] Determining a sensitive parameter group and a non-sensitive parameter group in the associated parameter group according to a change in a performance indicator value corresponding to the associated parameter group in each numerical adjustment includes:

[0025] Determine, according to the loss value corresponding to each numerical adjustment of the associated parameter group, a change value of the loss value corresponding to each numerical adjustment of the associated parameter group;

[0026] A sensitive parameter group and a non-sensitive parameter group are determined in the associated parameter group according to the loss value change value.

[0027] Optionally, the calculating the text similarity between the generated answer text and the reference answer text includes:

[0028] Converting the generated answer text and the reference answer text into a generated answer vector and a reference answer vector respectively;

[0029] The vector similarity between the generated answer vector and the reference answer vector is calculated, and the vector similarity is used as the text similarity.

[0030] The present invention also provides a text generation method, comprising:

[0031] Obtaining a pre-trained language model; the weight parameters to be trained in the pre-trained language model have been trained, and the weight parameters to be trained are determined using the parameter selection method as described above;

[0032] Receive an input user question text, input the user question text into the pre-trained language model to obtain a user answer text, and output the user answer text.

[0033] The present invention also provides a parameter selection device, comprising:

[0034] An acquisition module, used to acquire a benchmark question-answer text pair and a pre-trained language model, and extract weight parameters from the pre-trained language model;

[0035] An association analysis module, used for performing association analysis on the weight parameters to obtain association relationships between the weight parameters, and dividing the weight parameters into a plurality of association parameter groups according to the association relationships;

[0036] a model testing module, for adjusting the values ​​of the associated parameter groups in the pre-trained language model for each associated parameter group multiple times, and determining the performance index value of the pre-trained language model in a preset performance index using the benchmark question-answer text pair when each value adjustment is completed;

[0037] A parameter group classification module is used to determine a sensitive parameter group and a non-sensitive parameter group in the associated parameter group according to the change of the performance indicator value corresponding to the associated parameter group in each numerical adjustment, so as to determine the weight parameters in the sensitive parameter group as the weight parameters to be trained.

[0038] The present invention also provides a text generation device, comprising:

[0039] A model acquisition module, used to acquire a pre-trained language model; the weight parameters to be trained in the pre-trained language model have been trained, and the weight parameters to be trained are determined using the parameter selection method as described above;

[0040] The online service module is used to receive an input user question text, input the user question text into the pre-trained language model to obtain a user answer text, and output the user answer text.

[0041] The present invention also provides an electronic device, comprising:

[0042] Memory for storing computer programs;

[0043] A processor is used to implement the parameter selection method and / or the text generation method as described above when executing the computer program.

[0044] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the parameter selection method as described above and / or the text generation method as described above are implemented.

[0045] The present invention provides a parameter selection method, comprising: obtaining a benchmark question-answer text pair and a pre-trained language model, and extracting weight parameters from the pre-trained language model; performing correlation analysis on the weight parameters to obtain correlation relationships between the weight parameters, and dividing the weight parameters into multiple correlation parameter groups according to the correlation relationships; for each correlation parameter group, performing multiple numerical adjustments on the correlation parameter group in the pre-trained language model, and determining the performance index value obtained by the pre-trained language model in a preset performance index when each numerical adjustment is completed using the benchmark question-answer text pair; and determining a sensitive parameter group and a non-sensitive parameter group in the correlation parameter group according to changes in the performance index value corresponding to the correlation parameter group in each numerical adjustment, so as to determine the weight parameters in the sensitive parameter group as the weight parameters to be trained.

[0046] It can be seen that the present invention can first obtain a benchmark question-answer text pair and a pre-trained language model, and extract weight parameters from the model. Among them, the benchmark question-answer text pair can be preset manually, including preset text generation requirement information. Subsequently, the present invention can perform a correlation analysis on each weight parameter to obtain the correlation relationship between each weight parameter, and then the weight parameter can be divided into multiple association parameter groups according to the correlation relationship. Subsequently, for each association parameter group, the present invention makes multiple numerical adjustments to the association parameter group in the pre-trained language model, and uses the benchmark question-answer text pair to determine the performance index value obtained by the pre-trained language model in the preset performance index when each numerical adjustment is completed, so that the performance impact of each group of association parameter groups on the pre-trained language model to the benchmark question-answer text pair can be tested, and according to the change of the performance index value corresponding to the association parameter group in each numerical adjustment, the sensitive parameter group and the non-sensitive parameter group can be determined in the association parameter group, wherein the sensitive parameter group is the association parameter group that significantly affects the model performance, and the non-sensitive parameter group is the association parameter group that slightly affects the model performance. Furthermore, the present invention can automatically analyze the correlation of each weight parameter in the pre-trained language model and its contribution to model reasoning, and automatically determine the weight parameters to be trained, thereby improving the efficiency of further training of the pre-trained language model by relevant personnel. The present invention also provides a text generation method, a parameter selection device, a text generation device, an electronic device, and a computer-readable storage medium, which have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A flow chart of a parameter selection method provided by an embodiment of the present invention;

[0049] Figure 2 A flowchart of a text generation method provided by an embodiment of the present invention;

[0050] Figure 3 A flow chart of weight analysis provided by an embodiment of the present invention;

[0051] Figure 4 A flow chart of a weight parameter analysis provided by an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of a visual display provided by an embodiment of the present invention;

[0053] Figure 6 A flow chart of a sensitivity analysis provided by an embodiment of the present invention;

[0054] Figure 7 A structural block diagram of a parameter selection device provided by an embodiment of the present invention;

[0055] Figure 8 A structural block diagram of a text generation device provided by an embodiment of the present invention;

[0056] Fig. 9 The present invention provides a structural block diagram of an electronic device. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] A pre-trained language model is a machine learning model that has been trained with a large amount of text data, and is mainly used to parse and generate natural language text. Common application scenarios of this model are: receiving question text input by the user, parsing user needs from the question text, and generating output text that meets user needs. In the related art, the pre-trained language model needs to be fine-tuned to meet the needs of specific application scenarios, and the fine-tuning training requires relevant personnel to decide which part of the weight parameters in the model to update. However, the related art lacks the corresponding means to intuitively and comprehensively parse the weights of the deep learning model, which is not easy to understand the impact of the weight parameters on the model performance, and it is easy to affect the choice of relevant personnel to update the weight parameters. For example, it is difficult for relevant personnel to fine-tune the appropriate weight parameters so that the trained model can better output output text that meets user needs.

[0059] In view of this, in order to solve the technical problem of how to select key weight parameters to be optimized from a pre-trained language model to significantly improve the performance of the pre-trained language model in meeting user needs, the present invention can provide a parameter selection method, which can automatically analyze the correlation between each weight parameter in the pre-trained language model and its contribution to model reasoning, and automatically select the weight parameters to be trained, thereby improving the efficiency of relevant personnel in further training the pre-trained language model.

[0060] Please refer to Figure 1 , Figure 1 A flow chart of a parameter selection method provided by an embodiment of the present invention, the method may include:

[0061] S101. Obtain a benchmark question-answer text pair and a pre-trained language model, and extract weight parameters from the pre-trained language model.

[0062] In this step, first, a benchmark question-answer text pair and a pre-trained language model can be obtained. Among them, the benchmark question-answer text pair is a manually preset text pair, which may include a question text and a benchmark answer text. It is worth noting that the benchmark answer text pair contains demand information. For example, the question text may include a specific demand text; for another example, the benchmark answer text has an artificially specified language style. This embodiment does not limit the content contained in the benchmark question-answer text pair, for example, it may include an original text summary question-answer pair and an expanded question-answer pair, wherein the question text of the original text summary question-answer pair is the original text to be summarized, and the benchmark answer text is the summarized text; the question text of the expanded question-answer pair is the content and text style of the text to be expanded, and the benchmark answer text is the expanded text. The pre-trained language model refers to a machine learning model that has been trained with a large amount of text data and has basic text generation capabilities. After training and fine-tuning the benchmark question-answer text pair, the pre-trained language model has online service capabilities. The focus of this embodiment is on how to determine from the model the key weight parameters that significantly affect its corresponding performance when dealing with the above-mentioned benchmark question-answer text pairs, so as to conduct targeted training on the key weight parameters.

[0063] Furthermore, this step also needs to extract weight parameters from the pre-trained language model, and the weight parameters come from each network layer in the model, such as the convolution layer, the fully connected layer, the decoding layer, etc. It is worth noting that since the convolution layer, the fully connected layer, and the decoding layer are the core parts of the pre-trained language model, the weight parameters obtained in this embodiment can only come from these three network layers. Of course, this embodiment can also obtain the training log of the model, and determine the special training parameters that significantly affect the training results of the model through the training log, so as to obtain these special weight parameters.

[0064] Furthermore, in addition to obtaining the weight parameters, this embodiment can also determine the numerical value, type (such as whether it is a bias parameter), precision, etc. of each weight parameter to facilitate subsequent analysis.

[0065] S102: Perform correlation analysis on the weight parameters to obtain correlation relationships between the weight parameters, and divide the weight parameters into multiple correlation parameter groups according to the correlation relationships.

[0066] In this step, the present embodiment can perform a correlation analysis on the acquired weight parameters to obtain a correlation relationship between the weight parameters, and the correlation relationship indicates that multiple weight parameters are correlated or uncorrelated with each other. The purpose of performing this correlation analysis is that the pre-trained language model has been trained with a large amount of text data, and the weight parameters contained therein have formed a correlation relationship in multiple rounds of training, and this correlation relationship ensures that the model can achieve better results when performing conventional text generation tasks. Considering that the task represented by the benchmark question and answer text pair in this embodiment is also a text generation task, these weight parameters with a correlation relationship have a potential contribution to optimizing the model and improving the adaptability of the model when dealing with the benchmark question and answer text pair. Therefore, the present embodiment can pre-screen these weight parameters with correlation to improve the effectiveness and reliability of the selection of the weight parameters to be trained.

[0067] It should be noted that the above-mentioned association relationship can be either an association relationship between two weight parameters or an association relationship between multiple weight parameters (the number is greater than or equal to 2). This embodiment does not limit the specific association analysis method, which can be set according to actual application requirements. For example, for the association relationship between two weight parameters, it can be determined by calculating the correlation coefficient between the weights and comparing the correlation coefficient with a preset threshold. For the association relationship between multiple weight parameters, all weight parameters can also be clustered according to the values ​​of the weight parameters to obtain multiple parameter clustering categories, and determined according to the parameter clustering categories. The above two means can be used either to flexibly meet various parameter association analysis requirements.

[0068] Based on this, the performing of correlation analysis on the weight parameters to obtain the correlation relationship between the weight parameters, and dividing the weight parameters into a plurality of correlation parameter groups according to the correlation relationship, may include:

[0069] Step 11: Calculate the correlation coefficient between the weight parameters;

[0070] Step 12: grouping a pair of weight parameters whose correlation coefficient is greater than a preset threshold into an associated parameter group;

[0071] Or, step 21: clustering all the weight parameters according to the values ​​of the weight parameters to obtain multiple parameter clustering categories;

[0072] Step 22: Divide the weight parameters belonging to the same parameter clustering category into an associated parameter group.

[0073] It should be noted that this embodiment does not limit how to calculate the correlation coefficient between the weight parameters, and can be set according to actual application requirements. For example, the correlation coefficient can be a Pearson correlation coefficient, which is calculated as follows:

[0074] Step 31: Calculate the covariance and standard deviation between the first weight parameter and the second weight parameter;

[0075] Step 32: taking the ratio of the covariance to the standard deviation as the correlation coefficient between the first weight parameter and the second weight parameter.

[0076] The Pearson correlation coefficient can be expressed as:

[0077] ;

[0078] Among them, Cov(X, Y) represents the covariance of weight parameters X and Y, and σX and σY represent the standard deviations of weights X and Y. By calculating the covariance and standard deviation, the correlation coefficient between the two weights can be obtained.

[0079] This embodiment also does not limit how to perform clustering processing on the weight parameters. For example, K-means clustering (K-means) or a hierarchical clustering algorithm may be applied to cluster and group the weights.

[0080] S103. For each associated parameter group, perform multiple numerical adjustments on the associated parameter group in the pre-trained language model, and use the benchmark question-answer text pair to determine the performance indicator value achieved by the pre-trained language model in a preset performance indicator after each numerical adjustment is completed.

[0081] This step will test the performance changes brought about by adjusting each associated parameter group to the pre-trained language model in response to the above-mentioned benchmark question-answer text pair. Specifically, this embodiment needs to make multiple numerical adjustments to the associated parameter group in the pre-trained language model, and use the benchmark question-answer text pair to determine the performance indicator value achieved by the pre-trained language model in the preset performance indicator when each numerical adjustment is completed. For the numerical adjustment, this embodiment can preset a numerical adjustment strategy, which includes any one or more of a numerical adjustment range, a numerical adjustment step, or a random adjustment strategy. Among them, the numerical adjustment step represents the range of each numerical adjustment, and the random adjustment strategy is used to determine whether to perform a random numerical adjustment. In this way, this embodiment can adjust each weight parameter in the associated parameter group more standardized according to the preset numerical adjustment strategy.

[0082] Based on this, multiple numerical adjustments are made to the associated parameter group in the pre-trained language model, including:

[0083] Step 41: performing multiple numerical adjustments on the associated parameter group in the pre-trained language model according to a preset numerical adjustment strategy; the preset numerical adjustment strategy includes any one or more of a numerical adjustment range, a numerical adjustment step or a random adjustment strategy.

[0084] Furthermore, it should be noted that the present embodiment does not limit specific performance indicator values, which can be set according to actual application requirements, for example, it can be a model loss value, or it can be the accuracy of the model answering the question text. Taking the loss value as an example, the following specifically introduces the process of using the benchmark question-answer text pair to determine the performance indicator value obtained by the pre-trained language model in the preset performance indicator. Based on this, the benchmark question-answer text pair includes a question text and a benchmark answer text; the method of using the question-answer text pair to determine the performance indicator value obtained by the pre-trained language model in the preset performance indicator when each numerical adjustment is completed may include:

[0085] Step 51: After each numerical adjustment is completed, the question text is input into the pre-trained language model to obtain a generated answer text.

[0086] In this step, each time the numerical adjustment is completed, the present embodiment can input the question text into the pre-trained language model so that the model parses the question text according to the adjusted model parameters and outputs a generated answer text.

[0087] Step 52: Calculate the text similarity between the generated answer text and the benchmark answer text.

[0088] In this step, the difference between the generated answer text and the reference answer text is calculated by this embodiment. The greater the text similarity, the smaller the difference between the generated answer text and the reference answer text; the smaller the text similarity, the greater the difference between the generated answer text and the reference answer text. Based on the text similarity, this embodiment can determine the performance change of the pre-trained language model caused by parameter adjustment.

[0089] It should be noted that this embodiment does not limit how to determine the text similarity between the generated answer text and the reference answer text. For example, the generated answer text and the reference answer text can be converted into a generated answer vector and a reference answer vector respectively, and the vector similarity between the generated answer vector and the reference answer vector is calculated to use the vector similarity as the text similarity. In this way, the similarity between the generated answer text and the reference answer text can be determined based on the vector encoding form.

[0090] Based on this, the calculating of the text similarity between the generated answer text and the reference answer text may include:

[0091] Step 61: Convert the generated answer text and the reference answer text into a generated answer vector and a reference answer vector respectively;

[0092] Step 62: Calculate the vector similarity between the generative answer vector and the reference answer vector, and use the vector similarity as the text similarity.

[0093] Of course, the text similarity between the generated answer text and the benchmark answer file can also be determined based on the vocabulary usage similarity between the two, which can be set according to actual application requirements.

[0094] Step 52: Determine the loss value of the pre-trained language model in this numerical adjustment according to the text similarity.

[0095] In this step, the loss value is negatively correlated with the text similarity, that is, the greater the text similarity, the smaller the loss value. This embodiment does not limit the specific method of converting the text similarity into the loss value, which can be set according to actual application requirements.

[0096] S104. Determine a sensitive parameter group and a non-sensitive parameter group in the associated parameter group according to changes in the performance indicator values ​​corresponding to the associated parameter group in each numerical adjustment, so as to determine the weight parameters in the sensitive parameter group as the weight parameters to be trained.

[0097] In this step, after obtaining the performance index value generated by adjusting each group of associated parameter groups, this embodiment can determine the sensitive parameter group and the insensitive parameter group in the associated parameter group according to the change of the performance index value, so as to train the weight parameters in the sensitive parameter group. Among them, the sensitive parameter group refers to the associated parameter group whose numerical adjustment will significantly affect the model performance, and the insensitive parameter group refers to the associated parameter group whose numerical adjustment only slightly affects the model performance. Taking the loss value as an example, this embodiment can determine the loss value change value corresponding to the associated parameter group between each numerical adjustment according to the loss value caused by the associated parameter group in each numerical adjustment. Subsequently, the sensitive parameter group and the insensitive parameter group can be determined in the associated parameter group according to the loss value change value. For example, the associated parameter group whose loss value change value is greater than the preset threshold value can be determined as the sensitive parameter group, and the associated parameter group whose loss value change value is not greater than the preset threshold value can be determined as the insensitive parameter group; the associated parameter group whose loss value change rate is greater than the preset threshold value can also be determined as the sensitive parameter group, and the associated parameter group whose loss value change rate is not greater than the preset threshold value can be determined as the insensitive parameter group.

[0098] Based on this, determining the sensitive parameter group and the non-sensitive parameter group in the associated parameter group according to the change of the performance indicator value corresponding to each value adjustment of the associated parameter group may include:

[0099] Step 61: determining a change value of the loss value corresponding to each numerical adjustment of the associated parameter group according to the loss value corresponding to each numerical adjustment of the associated parameter group;

[0100] Step 62: Determine a sensitive parameter group and a non-sensitive parameter group in the associated parameter group according to the loss value change value.

[0101] Furthermore, in order to facilitate relevant personnel to view the automatic analysis results, the present embodiment may also use a variety of chart generation methods to convert the correlation between weight parameters, the change of performance indicator values ​​of sensitive parameter groups, and the change of performance indicator values ​​of non-sensitive parameter groups into a variety of visualization charts, and output and display these visualization charts. The present embodiment does not limit the specific icon generation method, for example, it can be a histogram, a heat map, a scatter plot, etc.

[0102] Based on this, after determining the sensitive parameter group and the non-sensitive parameter group in the associated parameter group, the method may further include:

[0103] Step 71: converting the correlation between the weight parameters, the change of the performance indicator value of the sensitive parameter group, and the change of the performance indicator value of the non-sensitive parameter group into a plurality of visual charts by using a plurality of chart generation methods;

[0104] Step 72: Output the visual chart to a display device.

[0105] In addition, to facilitate relevant personnel to analyze the model structure, this embodiment can also extract the model structure parameters of the pre-trained language model, and generate a model structure schematic diagram of the pre-trained language model based on the model structure parameters, and then output the model structure schematic diagram to the display device.

[0106] Based on this, it can also include:

[0107] Step 81: extracting model structure parameters of the pre-trained language model, and generating a model structure schematic diagram of the pre-trained language model according to the model structure parameters;

[0108] Step 82: Output the model structure schematic diagram to the display device.

[0109] Based on the above embodiments, the present invention can first obtain a benchmark question-answer text pair and a pre-trained language model, and extract weight parameters from the model. Among them, the benchmark question-answer text pair can be preset manually, including preset text generation requirement information. Subsequently, the present invention can perform a correlation analysis on each weight parameter to obtain the correlation relationship between each weight parameter, and then divide the weight parameter into multiple association parameter groups according to the correlation relationship. Subsequently, for each association parameter group, the present invention performs multiple numerical adjustments on the association parameter group in the pre-trained language model, and uses the benchmark question-answer text pair to determine the performance index value obtained by the pre-trained language model in the preset performance index when each numerical adjustment is completed, so as to test the performance impact of each group of association parameter groups on the pre-trained language model to the benchmark question-answer text pair, and can determine the sensitive parameter group and the non-sensitive parameter group in the association parameter group according to the change of the performance index value corresponding to the association parameter group in each numerical adjustment, wherein the sensitive parameter group is the association parameter group that significantly affects the model performance, and the non-sensitive parameter group is the association parameter group that slightly affects the model performance. Furthermore, the present invention can automatically analyze the correlation between each weight parameter in the pre-trained language model and its contribution to model reasoning, and automatically determine the weight parameters to be trained, thereby improving the efficiency of relevant personnel in further training the pre-trained language model.

[0110] Based on the above embodiment, a text generation method provided by an embodiment of the present invention is introduced below. Figure 2 , Figure 2 A flowchart of a text generation method provided by an embodiment of the present invention, the method may include:

[0111] S201, obtaining a pre-trained language model; the weight parameters to be trained in the pre-trained language model have been trained, and the weight parameters to be trained are determined using the parameter selection method as described above.

[0112] The pre-trained language model obtained in this step has been processed with parameters according to the parameter selection method described above to obtain weight parameters to be trained, and the above-mentioned benchmark question and answer text has been used to train and optimize this batch of weight parameters to be trained. This embodiment does not limit the specific process of weight parameter training and optimization, and reference may be made to related technologies.

[0113] Of course, since there can be multiple groups of sensitive parameter groups, that is, there can be multiple batches of weight parameters to be trained. If after training the current batch of weight parameters to be trained in the pre-trained language model, it is determined that the performance of the model is not improved well, the model can be rolled back to the previous version and trained based on the next group of weight parameters to be trained.

[0114] S202, receiving an input user question text, inputting the user question text into the pre-trained language model to obtain a user answer text, and outputting the user answer text.

[0115] As described above, the focus of this embodiment is how to determine from the model the key weight parameters that significantly affect its corresponding performance when dealing with the above-mentioned benchmark question-answer text pairs, so as to conduct targeted training on the key weight parameters. Therefore, after completing the screening of the sensitive parameter group and training the pre-trained language model based on the sensitive parameter group to obtain a model with improved performance, this embodiment can ensure that the model can output high-quality answer text when receiving a user question text with the same requirements as the benchmark answer text.

[0116] Based on the above embodiments, the parameter selection method and text generation method are fully introduced below based on specific schematic diagrams. The present invention provides a special device, which may include a weight parsing module, a visualization module and a model optimization suggestion module. The weight parsing module is responsible for obtaining the weight of the large language model and analyzing the weight parameters of the large language model. The visualization module generates an intuitive model weight map based on the parsing results, and the model optimization suggestion module provides targeted model optimization suggestions based on the map. Please refer to Figure 3 , Figure 3 A flowchart of weight analysis provided by an embodiment of the invention. The present invention constructs a large language model weight analysis framework, combines statistical analysis and visualization technology, and analyzes the model weight from multiple angles and levels. The method specifically includes the following steps:

[0117] S1. Collect and organize weight data during large language model training.

[0118] S2. Use statistical analysis methods to conduct in-depth research on the distribution, changing trends and correlations of weight data.

[0119] S3. Use visualization technology to intuitively display the statistical analysis results, including but not limited to weight heat maps, weight change curves, etc.

[0120] S4. Tune the model weights based on the analysis results to improve the performance and interpretability of the model.

[0121] In step S1, the present invention can use a deep learning framework to pre-train the large language model, obtain the pre-trained model and the training log file, or directly download the pre-trained model. Currently, the pre-trained model includes two mainstream formats: model.bin and model.safetensors.

[0122] In step S2, each weight parameter in the model needs to be analyzed one by one. Please refer to Figure 4 , Figure 4 A flow chart of a weight parameter analysis provided by an embodiment of the present invention. The analysis of weight data includes the following contents:

[0123] S21. Get all the weight parameters of the model. Use a specific library or tool to get all the weight parameters of the model. Count the weight parameters of each layer of the large language model. You can use tools such as Pytorch to build an automated analysis framework to analyze the weights of each layer of the large language model and whether there is bias. Print the accuracy and dimensionality information of the weights used.

[0124] Here we need to focus on the convolutional layer (conv.weight\conv.bias), fully connected layer (fc.weight\fc.bias), and transformer-encoder layer of the large language model. These layers are generally the core parts of the large language model. If the weights are obtained through training, you can analyze the model training log file saved in S1 to understand its training process and optimization goals.

[0125] S22. Model weight correlation analysis uses the Pearson correlation coefficient to analyze the correlation between different weights, which can identify which weights show similar trends in value, and thus infer the correlation between them. Principal Component Analysis (PCA) can also be used to identify the weight combinations that have the greatest impact on the model, as well as the correlation between them.

[0126] Then, K-means clustering or hierarchical clustering algorithm is comprehensively applied to group the weights. Through clustering, weights with high correlation can be classified into one category, and then their common characteristics and influencing factors can be analyzed, so as to have a more comprehensive understanding of the correlation of model weights and its impact on model performance.

[0127] S23. Export the pre-trained large language model to ONNX format. You can use third-party software to view the structure of the ONNX model and the weight information of the operators in the model, and perform statistical analysis of the model weights.

[0128] Step S3 builds an interactive visualization interface to display weight distribution, correlation and dynamic changes, allowing users to freely explore and analyze data as needed. Figure 5 , Figure 5 A schematic diagram of a visual display provided by an embodiment of the present invention. The visualization techniques and methods that can be selected in this method are as follows:

[0129] S31. The distribution of weight values ​​can be analyzed based on the weight distribution histogram, and the frequency of occurrence of different weight values ​​can be clearly seen. By adjusting the group distance and number of groups of the histogram, more detailed distribution information can be obtained.

[0130] S32. Using heat maps or scatter plot matrices to visualize the correlation between weights can intuitively show the degree of correlation between weights.

[0131] S33. Use the kernel density estimation method to smoothly display the continuous distribution of weights. This method can provide a more intuitive visual effect and help understand the overall distribution shape of weights.

[0132] S34. Plotting the Cumulative Distribution Function (CDF) of the weights can show the cumulative probability of the weights at different thresholds. This helps to understand the skewness and kurtosis of the weight distribution and identify outliers.

[0133] S35. Use box plots to display the median, quartiles, maximum, minimum, and outliers of the weights. This helps to quickly identify the central tendency and dispersion of the weights.

[0134] S36. Analyze different quantiles of weights (such as the five-number summary: minimum, first quartile, median, third quartile, maximum) to fully understand the distribution of weights.

[0135] For the tuning of model weights in step S4, the present invention proposes a weight optimization algorithm based on sensitivity analysis, which can accurately identify weights that have a significant impact on model performance and perform targeted optimization. Figure 6 , Figure 6 A flow chart of a sensitivity analysis provided by an embodiment of the present invention. The following are general steps for weight optimization of sensitivity analysis:

[0136] S41. Determine the key parameters and use sensitivity analysis to determine which parameters the model output is most sensitive to. These parameters are the key parameters and have the greatest impact on the model output. The specific steps of sensitivity analysis are as follows:

[0137] S411. First, clarify the specific goal of sensitivity analysis, whether it is to evaluate the impact of changes in a single weight on the overall model, or to analyze the combined effects when multiple weights change simultaneously.

[0138] S412. Collect all the training text data required for the model and ensure the accuracy and completeness of the data. Pre-process the data, such as cleaning, formatting and standardization, for subsequent analysis.

[0139] S413. Build a test model based on the pre-trained text model to ensure that the model can accurately reflect the essence of the research problem.

[0140] S414. Design a strategy for weight changes, including the range, step size, and method of change. For example, you can use a method of gradually increasing or decreasing weights, or randomly adjust weights according to a certain distribution (such as normal distribution).

[0141] S415. Define indicators for evaluating model sensitivity, such as the rate of change of accuracy, the amount of change in loss function, etc. These indicators should be able to quantitatively reflect the sensitivity of model performance to changes in weights.

[0142] S416. According to the designed weight change strategy, gradually adjust the weights in the model and retrain the model. Record the model performance indicators after each adjustment.

[0143] S417. Analyze the changing trend of model performance indicators as weights change, identify sensitive weights and non-sensitive weights. Use visualization tools in S3 (such as box plots, scatter plots, or heat maps) to intuitively display the analysis results.

[0144] S42. Optimize key parameters based on the correlation analysis results in S22, such as adjusting the value range of sensitive weights, improving the model structure, or introducing regularization terms, to minimize the uncertainty and error of the model output. This usually involves adjusting the value range, weights, and other factors of key parameters.

[0145] S43. Verify the optimization effect. After optimizing key parameters, rerun the model to check whether the model output has improved. If it has improved, it means that the optimization is effective; if it has not improved or has become worse, the optimization process needs to be rechecked.

[0146] In addition to sensitivity analysis, there are other weight optimization methods, such as stochastic gradient descent, mini-batch gradient descent, and Adam optimizer, etc. These methods have their own advantages and disadvantages in practical applications, and which method to choose depends on the specific needs and circumstances of the specific problem.

[0147] In short, sensitivity analysis is an effective weight optimization algorithm, which analyzes the sensitivity of model output to parameters, determines key parameters, and then optimizes these parameters to achieve the purpose of weight optimization. However, the choice of which method to perform weight optimization still needs to be considered comprehensively according to the specific situation.

[0148] The parameter selection device, text generation device, electronic device, computer program product, and computer-readable storage medium provided in the embodiments of the present invention are introduced below. The parameter selection device, text generation device, electronic device, computer program product, and computer-readable storage medium described below can be referenced to each other with the model-assisted optimization method described above.

[0149] Please refer to Figure 7 , Figure 7A structural block diagram of a data processing device provided by an embodiment of the present invention, the device may include:

[0150] An acquisition module 701 is used to acquire a reference question-answer text pair and a pre-trained language model, and extract a weight parameter from the pre-trained language model;

[0151] The association analysis module 702 is used to perform association analysis on the weight parameters to obtain association relationships between the weight parameters, and divide the weight parameters into multiple association parameter groups according to the association relationships;

[0152] A model testing module 703 is used to adjust the values ​​of the associated parameter groups in the pre-trained language model for multiple times for each associated parameter group, and to determine the performance index value of the pre-trained language model in a preset performance index using the benchmark question-answer text pair when each value adjustment is completed;

[0153] The parameter group classification module 704 is used to determine a sensitive parameter group and a non-sensitive parameter group in the associated parameter group according to the change of the performance indicator value corresponding to the associated parameter group in each numerical adjustment, so as to determine the weight parameters in the sensitive parameter group as the weight parameters to be trained.

[0154] Optionally, the association analysis module 702 may include:

[0155] A correlation coefficient calculation submodule, used to calculate the correlation coefficient between the weight parameters;

[0156] A first division submodule is used to divide a pair of weight parameters whose correlation coefficient is greater than a preset threshold into an associated parameter group;

[0157] Or, a clustering submodule, used for clustering all the weight parameters according to the values ​​of the weight parameters to obtain a plurality of parameter clustering categories;

[0158] The second division submodule is used to divide the weight parameters belonging to the same parameter clustering category into an associated parameter group.

[0159] Optionally, the model testing module 703 includes:

[0160] An adjustment submodule is used to perform multiple numerical adjustments on the associated parameter group in the pre-trained language model according to a preset numerical adjustment strategy; the preset numerical adjustment strategy includes any one or more of a numerical adjustment range, a numerical adjustment step or a random adjustment strategy.

[0161] Optionally, the device may further include:

[0162] A chart conversion module, used to convert the correlation between the weight parameters, the change of the performance indicator value of the sensitive parameter group, and the change of the performance indicator value of the non-sensitive parameter group into a variety of visual charts by using a variety of chart generation methods;

[0163] The output display module is used to output the visual chart to a display device.

[0164] Optionally, the reference question-answer text pair includes a question text and a reference answer text;

[0165] The model testing module 703 includes:

[0166] The reasoning submodule is used to input the question text into the pre-trained language model to obtain a generated answer text each time the value adjustment is completed; and calculate the text similarity between the generated answer text and the reference answer text;

[0167] A loss value calculation submodule, used to determine the loss value generated by the pre-trained language model in this numerical adjustment according to the text similarity;

[0168] The parameter group classification module 704 includes:

[0169] A loss value change value calculation submodule, used to determine the loss value change value corresponding to each numerical adjustment of the associated parameter group according to the loss value corresponding to each numerical adjustment of the associated parameter group;

[0170] The grouping submodule is used to determine a sensitive parameter group and a non-sensitive parameter group in the associated parameter group according to the loss value change value.

[0171] Optionally, the loss value calculation submodule includes:

[0172] A vector conversion unit, used to convert the generative answer text and the reference answer text into a generative answer vector and a reference answer vector respectively;

[0173] The similarity calculation unit is used to calculate the vector similarity between the generative answer vector and the reference answer vector, and use the vector similarity as the text similarity.

[0174] Please refer to Figure 8 , Figure 8 A structural block diagram of a text generation device provided by an embodiment of the present invention, the device may include:

[0175] The model acquisition module 801 is used to obtain a benchmark question-answer text pair and a pre-trained language model; the weight parameters to be trained in the pre-trained language model have been trained, and the weight parameters to be trained are determined using the parameter selection method as described above; the online service module 802 is used to receive an input user question text, input the user question text into the pre-trained language model to obtain a user answer text, and output the user answer text.

[0176] It should be pointed out that the text generation device may include a parameter selection module, and the structure in the parameter selection module may be set according to the parameter selection device. For example, the parameter selection module may include an acquisition module, an association analysis module, a model testing module, a parameter group classification module, etc. in the parameter selection device.

[0177] Please refer to Fig. 9 , Fig. 9 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The embodiment of the present invention provides an electronic device 10, including a processor 11 and a memory 12; wherein the memory 12 is used to store a computer program; and the processor 11 is used to execute the parameter selection method and / or text generation method provided in the aforementioned embodiment when executing the computer program.

[0178] For the specific process of the above-mentioned parameter selection method and text generation method, reference may be made to the corresponding contents provided in the aforementioned embodiments, which will not be elaborated here.

[0179] Furthermore, the memory 12 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0180] In addition, the electronic device 10 also includes a power supply 13, a communication interface 14, an input / output interface 15 and a communication bus 16; wherein the power supply 13 is used to provide working voltage for each hardware device on the electronic device 10; the communication interface 14 can create a data transmission channel between the electronic device 10 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input / output interface 15 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0181] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the parameter selection method and / or text generation method described in the above embodiments.

[0182] Since the embodiments of the computer program product part correspond to the embodiments of the parameter selection method and text generation method part, please refer to the description of the embodiments of the parameter selection method and text generation method part for the embodiments of the computer program product part, and will not be repeated here.

[0183] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the parameter selection method and / or text generation method described in the above embodiment is implemented.

[0184] Since the embodiments of the computer-readable storage medium part correspond to the embodiments of the parameter selection method and text generation method part, the embodiments of the storage medium part refer to the description of the embodiments of the parameter selection method and text generation method part, which will not be repeated here.

[0185] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0186] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0187] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0188] The above is a detailed introduction to a parameter selection method, a text generation method, a device, an electronic device and a storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A parameter selection method, characterized in that: include: Obtaining a benchmark question-answer text pair and a pre-trained language model, and extracting weight parameters from the pre-trained language model; Performing correlation analysis on the weight parameters to obtain correlation relationships between the weight parameters, and dividing the weight parameters into a plurality of correlation parameter groups according to the correlation relationships; For each associated parameter group, multiple numerical adjustments are made to the associated parameter group in the pre-trained language model, and when each numerical adjustment is completed, the benchmark question-answer text pair is used to determine the performance indicator value obtained by the pre-trained language model in a preset performance indicator; According to the change of the performance indicator value corresponding to the associated parameter group in each numerical adjustment, a sensitive parameter group and a non-sensitive parameter group are determined in the associated parameter group, so as to determine the weight parameters in the sensitive parameter group as the weight parameters to be trained.

2. The parameter selection method according to claim 1, characterized in that: The performing correlation analysis on the weight parameters to obtain correlation relationships between the weight parameters, and dividing the weight parameters into a plurality of correlation parameter groups according to the correlation relationships, includes: Calculating the correlation coefficient between each of the weight parameters; Classify a pair of weight parameters whose correlation coefficient is greater than a preset threshold into an associated parameter group; Or, clustering all the weight parameters according to the values ​​of the weight parameters to obtain a plurality of parameter clustering categories; Weight parameters that belong to the same parameter cluster category are grouped into an associated parameter group.

3. The parameter selection method according to claim 1, characterized in that: The associated parameter group is adjusted multiple times in the pre-trained language model, including: The associated parameter group is adjusted multiple times in the pre-trained language model according to a preset numerical adjustment strategy; the preset numerical adjustment strategy includes any one or more of a numerical adjustment range, a numerical adjustment step or a random adjustment strategy.

4. The parameter selection method according to claim 1, characterized in that: After determining the sensitive parameter group and the non-sensitive parameter group in the associated parameter group, the method further includes: Using a variety of chart generation methods to convert the correlation between the weight parameters, the change of the performance indicator value of the sensitive parameter group, and the change of the performance indicator value of the non-sensitive parameter group into a variety of visual charts; The visual diagram is output to a display device.

5. The parameter selection method according to any one of claims 1 to 4, characterized in that: The reference question-answer text pair includes a question text and a reference answer text; The step of using the question-answer text pair to determine the performance indicator value obtained by the pre-trained language model in the preset performance indicator when each value adjustment is completed includes: When each value adjustment is completed, the question text is input into the pre-trained language model to obtain a generated answer text; Calculating text similarity between the generated answer text and the reference answer text; Determine the loss value of the pre-trained language model generated in this numerical adjustment according to the text similarity; Determining a sensitive parameter group and a non-sensitive parameter group in the associated parameter group according to a change in a performance indicator value corresponding to the associated parameter group in each numerical adjustment includes: Determine, according to the loss value corresponding to each numerical adjustment of the associated parameter group, a change value of the loss value corresponding to each numerical adjustment of the associated parameter group; A sensitive parameter group and a non-sensitive parameter group are determined in the associated parameter group according to the loss value change value.

6. A text generation method, characterized in that: include: Obtaining a pre-trained language model; the weight parameters to be trained in the pre-trained language model have been trained, and the weight parameters to be trained are determined using the parameter selection method according to any one of claims 1 to 5; Receive an input user question text, input the user question text into the pre-trained language model to obtain a user answer text, and output the user answer text.

7. A parameter selection device, characterized in that: include: An acquisition module, used to acquire a benchmark question-answer text pair and a pre-trained language model, and extract weight parameters from the pre-trained language model; An association analysis module, used for performing association analysis on the weight parameters to obtain association relationships between the weight parameters, and dividing the weight parameters into a plurality of association parameter groups according to the association relationships; a model testing module, for adjusting the values ​​of the associated parameter groups in the pre-trained language model for each associated parameter group multiple times, and determining the performance index value of the pre-trained language model in a preset performance index using the benchmark question-answer text pair when each value adjustment is completed; A parameter group classification module is used to determine a sensitive parameter group and a non-sensitive parameter group in the associated parameter group according to changes in the performance indicator values ​​corresponding to the associated parameter group in each numerical adjustment, so as to determine the weight parameters in the sensitive parameter group as the weight parameters to be trained.

8. A text generation device, characterized in that: include: A model acquisition module, used for acquiring a pre-trained language model; the weight parameters to be trained in the pre-trained language model have been trained, and the weight parameters to be trained are determined by the parameter selection method according to any one of claims 1 to 5; The online service module is used to receive an input user question text, input the user question text into the pre-trained language model to obtain a user answer text, and output the user answer text.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the parameter selection method as described in any one of claims 1 or 5 and / or the text generation method as described in claim 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the parameter selection method according to any one of claims 1 or 5 and / or the text generation method according to claim 6 are implemented.

Images