Natural language general model capability optimization evaluation method, system and device and storage medium

By performing correlation calculation and model training on the optimization data of natural language general models, the optimization combination is selected, and the problem of missing optimal combinations and excessive time-consuming is solved, and fast and accurate model optimization is achieved.

CN119046649APending Publication Date: 2024-11-29SI-TECH INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410934115.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-12
Publication Date
2024-11-29

AI Technical Summary

Technical Problem

In natural language general model intelligent response system, it is difficult to find all optimization combinations by random sampling or full-scale combination trials, especially when there are many types of tuning methods, resulting in missing optimal combinations, and full-scale combination trials take too long. Missed the validity period of the model results.

Method used

By obtaining the optimized model quality data, classifying and calculating the correlation of each optimization method, using the entropy weight TOPSIS and LightGBM tree algorithm models for training and prediction, filtering out the first N maximum predicted values ​​for optimization test, and if the results do not match, update Training data until it meets the real situation.

Benefits of technology

It is realized that the basic optimal tuning combination is found without full combination experiments, which improves the optimization efficiency and accuracy of the general natural language model, reduces the risk of missing the optimal combination, and avoids excessive time-consuming Missing the validity period of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119046649A_ABST
    Figure CN119046649A_ABST
Patent Text Reader

Abstract

The invention discloses a natural language general model capability optimization evaluation method, system and device, and a storage medium. The method comprises the following steps: obtaining tested and optimized model quality data; classifying the quality data, taking each optimization mode as an attribute field, and obtaining a reasoning quality value corresponding to each optimization mode; calculating the association degree of each attribute field based on the attribute fields and the corresponding reasoning quality values; if the field association degree is greater than or equal to a preset threshold value, remodeling each feature with high similarity until the field association degree of each attribute is smaller than the threshold value; inputting the field association degree smaller than a threshold value into a LightGBM tree algorithm model for training to obtain an optimized evaluation prediction value; the first N maximum prediction values are screened out; inputting the first N maximum predicted values into a natural language general model for optimization testing, and outputting a quality evaluation value; and if the quality evaluation value is not consistent, inputting the quality evaluation value into a LightGBM tree algorithm model for training until the quality evaluation value is consistent with a real situation. The method is more accurate in prediction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of optimization and evaluation of natural language universal models, and in particular to a method, system, device and storage medium for optimizing and evaluating the capability of a natural language universal model. Background Art

[0002] In the optimization of the intelligent response system based on the general model of natural language, different combinations of various tuning techniques will result in different improvements and reductions in model capabilities, thus affecting the accuracy and rationality of the response content. For example, random sampling is used to try combinations. For all the optimization combination schemes, multiple groups of schemes are sampled and selected for optimization testing based on certain manual experience, and then the best combination scheme is selected based on the quality of the results; all the combination schemes are tried, and all the optimization combination schemes are optimized and tested one by one, and then the best combination scheme is selected based on the quality of the results.

[0003] Although the random sampling method of combination trial can reduce the number of missed optimal combinations with the help of manual experience, sampling is not a test of all optimization combinations after all. Therefore, many better optimization combinations will always be missed, especially in the scenario where there are too many related combinations, missing the best optimization combination is particularly serious; the best combination will definitely not be missed by trying all combinations, but it is only applicable to the case where the total number of tuning methods is small. Otherwise, if the number of tuning methods or the number of specific methods of each tuning method is quite large, the entire tuning process will be extremely time-consuming. Finally, the validity period of the model results may be missed due to the long tuning time. Summary of the invention

[0004] In view of the shortcomings existing in the above problems, the present invention provides a method, system, device and storage medium for optimizing and evaluating the capability of a general natural language model.

[0005] To achieve the above object, the present invention provides a method for optimizing and evaluating the capability of a general natural language model, comprising:

[0006] Obtain the quality data of the model after testing and optimization;

[0007] The quality data is classified according to the optimization methods, each optimization method is used as an attribute field, and a reasoning quality value corresponding to each optimization method is obtained through testing;

[0008] Calculate the relevance of each attribute field based on the attribute field and the corresponding reasoning quality value;

[0009] If the field correlation is greater than or equal to a preset threshold, the entropy weight TOPSIS is used to reshape the features with high similarity until the correlation of each attribute field is less than the threshold;

[0010] Input the field association degree less than the threshold into the LightGBM tree algorithm model for training to obtain an optimized evaluation prediction value;

[0011] Sort the optimization evaluation prediction values ​​from large to small, and select the top N largest prediction values;

[0012] Inputting the top N maximum prediction values ​​into a general natural language model for optimization testing, and outputting a quality assessment value;

[0013] If the quality assessment value does not meet the requirements, the quality assessment value is used to replace the field association degree that is less than the threshold value and is input into the LightGBM tree algorithm model for training until the quality assessment value meets the actual situation.

[0014] Preferably, the quality data is classified according to optimization methods, and each optimization method includes as an attribute field:

[0015] Convert the enumeration class of each attribute field of the optimization method into a numerical value. The conversion formula is:

[0016] (x n -x min ) / (x max -x min );

[0017] Among them, X is the optimization method; n is the nth optimization method; x n is the current value, x min is the minimum value of the current attribute, x max The maximum value of the current attribute.

[0018] Preferably, the correlation degree of each attribute field is calculated based on the attribute field and the corresponding reasoning quality value, and the calculation formula is:

[0019]

[0020] Among them, x i ={x 1、 x2, x3, ..., x n} is the attribute field element; y i ={y1, y2, y3, ..., y n} is the inference quality value, and n is the number of attribute field elements.

[0021] The present invention also provides a natural language general model capability optimization evaluation system, comprising:

[0022] An acquisition module is used to obtain the quality data of the model after testing and optimization;

[0023] A classification module, used to classify the quality data according to optimization methods, with each optimization method as an attribute field, and test to obtain the inference quality value corresponding to each optimization method;

[0024] A calculation module, used for calculating the relevance of each attribute field based on the attribute field and the corresponding reasoning quality value;

[0025] A screening module is used for reshaping each feature with high similarity by using entropy weight TOPSIS when the field correlation is greater than or equal to a preset threshold, until the correlation of each attribute field is less than the threshold;

[0026] A training module, used for inputting the field association degree less than the threshold into the LightGBM tree algorithm model for training to obtain an optimized evaluation prediction value;

[0027] A sorting module is used to sort the optimization evaluation prediction values ​​from large to small and filter out the top N largest prediction values;

[0028] A testing module, used for inputting the first N maximum prediction values ​​into a natural language general model for optimization testing, and outputting a quality evaluation value;

[0029] The verification module is used to replace the field association degree less than the threshold with the quality assessment value if the quality assessment value does not meet the requirements and input the quality assessment value into the LightGBM tree algorithm model for training until the quality assessment value meets the actual situation.

[0030] Preferably, the quality data is classified according to optimization methods, and each optimization method includes as an attribute field:

[0031] Convert the enumeration class of each attribute field of the optimization method into a numerical value. The conversion formula is:

[0032] (x n -x min ) / (x max -x min );

[0033] Among them, X is the optimization method; n is the nth optimization method; x n is the current value, x min is the minimum value of the current attribute, x max The maximum value of the current attribute.

[0034] Preferably, the correlation degree of each attribute field is calculated based on the attribute field and the corresponding reasoning quality value, and the calculation formula is:

[0035]

[0036] Among them, xi ={x 1、 x2, x3, ..., x n} is the attribute field element; y i ={y1, y2, y3, ..., y n} is the inference quality value, and n is the number of attribute field elements.

[0037] The present invention also provides a device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0038] The present invention also provides a storage medium storing a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device executes the above method.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention trains the combined data of tuning-related methods of the natural language general model that have been actually tested, and predicts the estimated quality of the natural language general model of the full tuning combination, so as to select the one with the best quality after tuning for verification. If it is indeed better, the verification is terminated. Otherwise, the actual tested data is used to update the original training data to make the prediction model more precise and the inferred tuning method more accurate. Therefore, there is no need to run all the tuning combinations, and the basically optimal tuning combination can be obtained, so that the natural language general model can quickly obtain the optimal optimization result. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flow chart of the natural language general model capability optimization evaluation method of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Reference Figure 1 The present invention provides a method for optimizing and evaluating the capability of a general natural language model, comprising:

[0044] Obtain the quality data of the model after testing and optimization;

[0045] The quality data is classified according to the optimization method, each optimization method is used as an attribute field, and the inference quality value corresponding to each optimization method is obtained through testing;

[0046] Calculate the correlation degree of each attribute field based on the attribute field and the corresponding reasoning quality value;

[0047] If the field correlation is greater than or equal to the preset threshold, the entropy weight TOPSIS is used to reshape the features with high similarity until the correlation of each attribute field is less than the threshold;

[0048] The field associations that are less than the threshold are input into the LightGBM tree algorithm model for training to obtain the optimized evaluation prediction value;

[0049] Sort the optimization evaluation prediction values ​​from large to small, and select the top N largest prediction values;

[0050] Input the top N largest prediction values ​​into the natural language general model for optimization testing and output the quality assessment value;

[0051] If the quality assessment value does not meet the requirements, the quality assessment value will replace the field correlation less than the threshold and input into the LightGBM tree algorithm model for training until the quality assessment value meets the actual situation.

[0052] In this embodiment, the quality data is classified according to the optimization method, and each optimization method as an attribute field includes:

[0053] Convert the enumeration class of each attribute field of the optimization method into a numerical value. The conversion formula is:

[0054] (x n -x min ) / (x max -x min );

[0055] Among them, X is the optimization method; n is the nth optimization method; x n is the current value, x min is the minimum value of the current attribute, x max The maximum value of the current attribute.

[0056] Furthermore, the correlation degree of each attribute field is calculated based on the attribute field and the corresponding reasoning quality value. The calculation formula is:

[0057]

[0058] Among them, x i ={x 1、 x2, x3, ..., x n} is the attribute field element; y i={y1, y2, y3, ..., y n} is the inference quality value, and n is the number of attribute field elements.

[0059] Example

[0060] The possible optimization methods for the natural language large model are classified and the specific methods under each classification are sorted out. An optimization method under an optimization classification is used as an attribute field. The inference quality value corresponding to each combination of optimization methods is tested, and the data table is shown in Table 1.

[0061] Table 1

[0062]

[0063] Convert the enumeration class of each attribute field of optimization mode 1 to optimization mode n into a numeric type. The numeric type uses the maximum and minimum value formula: (x n -x min ) / (x max -x min ) to perform standardized (normalized) data preprocessing operations. n is the current value, x min is the minimum value of the current attribute, x max is the maximum value of the current attribute;

[0064] Then use the formula: Calculate the correlation of each feature field. Two attribute field elements x to be compared i ={x1,x2,x3,......,x n},y i ={y1,y2,y3,......,y n}, n is the number of elements in the attribute field;

[0065] Set an appropriate relevance threshold for attribute reshaping based on common experience.

[0066] For attribute field groups whose correlation is greater than or equal to the set threshold, entropy weight TOPSIS is used to reshape the features of each group with high similarity until the correlation between each attribute field is less than the set threshold.

[0067] For attribute field groups whose correlation is less than the set threshold, data fusion is not performed on the corresponding attribute fields.

[0068] Use the LightGBM tree algorithm model with parallel processing to test and evaluate the regression modeling of the processed data;

[0069] Use the established model to infer the optimization evaluation of the full-scale tuning combination and output the corresponding prediction value;

[0070] Filter out the best several or multiple Top n maximum predicted values;

[0071] According to the output top n optimal tuning mode combinations, actual natural language general model optimization is performed respectively, and the relevant actual model quality evaluation values ​​are output;

[0072] Determine whether the verification of the tuning results of the large natural language model is satisfactory and consistent with the actual situation.

[0073] If the prediction results do not quite match the actual situation, or if you are not satisfied with the tuning results after actual verification, replace the corresponding result quality prediction value of the original natural language large model with the actual tested results.

[0074] After replacing the original inference quality value with the actual tested inference quality value, the new result data is used to update the result data of the original training database. For example, the original predicted inference quality value of optimization method groups 4, 7, and 9 is different from the current tested inference quality value, so it needs to be replaced with the actual tested value. The updated data is shown in Table 2.

[0075] Table 2

[0076]

[0077]

[0078] Reuse the LightGBM tree algorithm model with parallel processing to perform regression modeling on the processed data for test evaluation;

[0079] If the prediction results are consistent with the actual situation and the user is satisfied with the tuning results after actual verification, the tuning of the current natural language general model will be optimized according to the current tuning combination.

[0080] The present invention also provides a natural language general model capability optimization evaluation system, comprising:

[0081] An acquisition module is used to obtain the quality data of the model after testing and optimization;

[0082] The classification module is used to classify the quality data according to the optimization method, with each optimization method as an attribute field, and test to obtain the inference quality value corresponding to each optimization method;

[0083] A calculation module, used to calculate the relevance of each attribute field based on the attribute field and the corresponding reasoning quality value;

[0084] The screening module is used to reshape the features with high similarity by using entropy weight TOPSIS when the field correlation is greater than or equal to the preset threshold, until the correlation of each attribute field is less than the threshold;

[0085] The training module is used to input the field association degree less than the threshold into the LightGBM tree algorithm model for training to obtain the optimized evaluation prediction value;

[0086] The sorting module is used to sort the optimization evaluation prediction values ​​from large to small and filter out the top N largest prediction values;

[0087] A test module is used to input the top N maximum prediction values ​​into the natural language general model for optimization testing and output a quality evaluation value;

[0088] The verification module is used to replace the field association degree less than the threshold with the quality assessment value if the quality assessment value does not meet the requirements and input it into the LightGBM tree algorithm model for training until the quality assessment value meets the actual situation.

[0089] In this embodiment, the quality data is classified according to the optimization method, and each optimization method as an attribute field includes:

[0090] Convert the enumeration class of each attribute field of the optimization method into a numerical value. The conversion formula is:

[0091] (x n -x min ) / (x max -x min );

[0092] Among them, X is the optimization method; n is the nth optimization method; x n is the current value, x min is the minimum value of the current attribute, x max The maximum value of the current attribute.

[0093] Furthermore, the correlation degree of each attribute field is calculated based on the attribute field and the corresponding reasoning quality value. The calculation formula is:

[0094]

[0095] Among them, x i ={x 1、 x2, x3, ..., x n} is the attribute field element; y i ={y1, y2, y3, ..., y n} is the inference quality value, and n is the number of attribute field elements.

[0096] The present invention also provides a device, comprising at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the above method.

[0097] The present invention also provides a storage medium storing a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the above method.

[0098] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for optimizing and evaluating the capability of a general natural language model, characterized in that: include: Obtain the quality data of the model after testing and optimization; The quality data is classified according to the optimization methods, each optimization method is used as an attribute field, and a reasoning quality value corresponding to each optimization method is obtained through testing; Calculate the relevance of each attribute field based on the attribute field and the corresponding reasoning quality value; If the field correlation is greater than or equal to a preset threshold, the entropy weight TOPSIS is used to reshape the features with high similarity until the correlation of each attribute field is less than the threshold; Input the field association degree less than the threshold into the LightGBM tree algorithm model for training to obtain an optimized evaluation prediction value; Sort the optimization evaluation prediction values ​​from large to small, and select the top N largest prediction values; Inputting the top N maximum prediction values ​​into a general natural language model for optimization testing, and outputting a quality assessment value; If the quality assessment value does not meet the requirements, the quality assessment value is used to replace the field association degree that is less than the threshold value and is input into the LightGBM tree algorithm model for training until the quality assessment value meets the actual situation.

2. The method for optimizing and evaluating the capability of a general natural language model according to claim 1, characterized in that: The quality data is classified according to optimization methods, and each optimization method includes as an attribute field: Convert the enumeration class of each attribute field of the optimization method into a numerical value. The conversion formula is: (x n -x min ) / (x max -x min ); Among them, X is the optimization method; n is the nth optimization method; x n is the current value, x min is the minimum value of the current attribute, x max The maximum value of the current attribute.

3. The method for optimizing and evaluating the ability of a general natural language model according to claim 2, characterized in that: The correlation degree of each attribute field is calculated based on the attribute field and the corresponding reasoning quality value, and the calculation formula is: Among them, x i ={x 1、 x2, x3, ..., x n } is the attribute field element; y i ={y1, y2, y3, ..., y n } is the inference quality value, and n is the number of attribute field elements.

4. A natural language general model capability optimization evaluation system, characterized in that: include: An acquisition module is used to obtain the quality data of the model after testing and optimization; A classification module, used to classify the quality data according to optimization methods, with each optimization method as an attribute field, and test to obtain the inference quality value corresponding to each optimization method; A calculation module, used for calculating the relevance of each attribute field based on the attribute field and the corresponding reasoning quality value; A screening module is used for reshaping each feature with high similarity by using entropy weight TOPSIS when the field correlation is greater than or equal to a preset threshold, until the correlation of each attribute field is less than the threshold; A training module, used for inputting the field association degree less than the threshold into the LightGBM tree algorithm model for training to obtain an optimized evaluation prediction value; A sorting module is used to sort the optimization evaluation prediction values ​​from large to small and filter out the top N largest prediction values; A testing module, used for inputting the first N maximum prediction values ​​into a natural language general model for optimization testing, and outputting a quality evaluation value; The verification module is used to replace the field association degree less than the threshold with the quality assessment value if the quality assessment value does not meet the requirements and input the quality assessment value into the LightGBM tree algorithm model for training until the quality assessment value meets the actual situation.

5. The natural language general model capability optimization evaluation system according to claim 4 is characterized in that: The quality data is classified according to optimization methods, and each optimization method includes as an attribute field: Convert the enumeration class of each attribute field of the optimization method into a numerical value. The conversion formula is: (x n -x min ) / (x max -x min ); Among them, X is the optimization method; n is the nth optimization method; x n is the current value, x min is the minimum value of the current attribute, x max The maximum value of the current attribute.

6. The natural language general model capability optimization and evaluation system according to claim 5 is characterized in that: The correlation degree of each attribute field is calculated based on the attribute field and the corresponding reasoning quality value, and the calculation formula is: Among them, x i ={x 1、 x2, x3, ..., x n } is the attribute field element; y i ={y1, y2, y3, ..., y n } is the inference quality value, and n is the number of attribute field elements.

7. A device, characterized in that: The method comprises at least one processing unit and at least one storage unit, wherein the storage unit stores a computer program, and when the program is executed by the processing unit, the processing unit executes the method according to any one of claims 1 to 3.

8. A storage medium, characterized in that: It stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device executes the method described in any one of claims 1 to 3.