System and method for evaluating generalization ability of AI large model, and electronic equipment

Abstract

The invention relates to the technical field of large model generalization ability evaluation, and particularly discloses an AI large model generalization ability evaluation system and method and an electronic device, and the system is provided with a field generalization ability value evaluation module, a task generalization ability value evaluation module and a generalization ability optimization module. Firstly, a field generalization ability value of the large model and a task generalization ability value of the large model are evaluated and are compared with a preset field generalization ability reference value and a task generalization ability reference value respectively, so that the generalization ability of the large model is optimized, and the large model can keep good generalization performance; the method can better adapt to changes in an actual application environment, improves the overall prediction accuracy of the large model, enhances the robustness of the large model, enables the large model to be more stable and reliable in actual application, and reduces wrong evaluation caused by incomplete data analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of large model generalization capability evaluation, and specifically to an evaluation system, method and electronic device for the generalization capability of a large AI model. Background Art

[0002] The current technical solutions for evaluating the generalization ability of large models are that if the model is on a dataset with extremely unbalanced category distribution, its performance on unseen domains or tasks may be poor, thus limiting the generalization ability of the model. In addition, different evaluation indicators may focus on different aspects of the model. When selecting evaluation indicators, multiple aspects are not considered comprehensively, which will make the obtained parameters unable to fully reflect the generalization ability of the model. When the generalization ability of the model is evaluated without considering the limitations of the model's structure, parameters, etc., it is possible that the model cannot fully adapt to the data and tasks in the required field.

[0003] For example, the invention patent with announcement number CN114780722B, the above application involves a domain generalization method combined with a domain-general language model, which belongs to the field of artificial intelligence transfer learning technology; the above application combines pre-trained language models and model pruning technology, first fine-tunes the pre-trained language model, and uses multiple source domain data to train the pre-trained language model. Based on the model obtained after fine-tuning, the domain invariant scores of the parameters in the model are calculated, and the parameters with low domain invariant scores are pruned. Finally, the pruned language model is retrained, and the generalization performance of the trained domain-general language model is tested on different data. The above application solves the problem of large learning variance in over-parameterized pre-trained language models. The domain-general language model based on the present invention is significantly better than the corresponding baseline model, and has achieved good results in generalization tasks in the field of natural language processing.

[0004] Combining the above technical solutions, it is found that with the continuous development of large model technology, it is necessary to organize more accurate, reliable and stable large model generalization ability evaluation technical solutions. However, most large model generalization ability evaluation technical solutions have a relatively single dimension for model generalization ability evaluation and cannot comprehensively obtain accurate data on model generalization ability, making it difficult for model generalization ability evaluation to achieve the expected results, resulting in deviations in the evaluation of large models, which ultimately affects the generalization ability of the model. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a system, method and electronic device for evaluating the generalization capability of a large AI model, which can effectively solve the problems involved in the above-mentioned background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: In the first aspect, the present invention provides an evaluation system for the generalization ability of an AI large model, including: a domain generalization ability value evaluation module, which is used to obtain the domain generalization data set of the large model in each test domain based on the construction of real scenarios, analyze the domain generalization performance indicators of the large model in each test domain, and evaluate the domain generalization ability value of the large model; a task generalization ability value evaluation module, which is used to obtain the task generalization data set of the large model in each test task based on the construction of real scenarios, analyze the task generalization performance indicators of the large model in each test task, and comprehensively evaluate the task generalization ability value of the large model; a generalization ability optimization module, which is used to compare the domain generalization ability value of the large model and the task generalization ability value of the large model with the preset domain generalization ability reference value and the task generalization ability reference value, respectively, so as to optimize the generalization ability of the large model.

[0007] As a further solution, the optimization of the generalization ability of the large model is achieved, and the specific analysis process is: compare the domain generalization ability value of the large model with the preset domain generalization ability reference value. If the domain generalization ability value of the large model is greater than or equal to the domain generalization ability reference value, there is no need to optimize the domain generalization ability of the large model. If the domain generalization ability value of the large model is less than the domain generalization ability reference value, the domain generalization ability value of the large model and the domain generalization ability reference value are differenced to obtain the domain generalization ability deviation value of the large model, and the domain generalization ability optimization plan corresponding to each domain generalization ability deviation value interval defined in the evaluation system control library is matched to obtain the domain generalization ability optimization plan of the large model, so as to optimize the domain of the large model. The generalization capability is optimized; the task generalization capability value of the large model is compared with the preset task generalization capability reference value. If the task generalization capability value of the large model is greater than or equal to the task generalization capability reference value, there is no need to optimize the task generalization capability of the large model. If the task generalization capability value of the large model is less than the task generalization capability reference value, the task generalization capability value of the large model is differenced with the task generalization capability reference value to obtain the task generalization capability deviation value of the large model, which is matched with the task generalization capability optimization plan corresponding to each task generalization capability deviation value interval defined in the evaluation system control library to obtain the task generalization capability optimization plan of the large model, so as to optimize the task generalization capability of the large model and finally achieve the optimization of the generalization capability of the large model.

[0008] As a further solution, the domain generalization capability value of the large model has the following specific analysis process: based on the domain generalization data set of the large model in each test domain, the domain generalization accuracy of the large model in each test domain and the domain generalization recall of the large model in each test domain are analyzed, and the domain generalization capability value of the large model is comprehensively determined by performing data processing with the preset domain generalization reference accuracy and domain generalization reference recall.

[0009] As a further solution, the task generalization capability value of the large model is specifically analyzed as follows: according to the task generalization data set of the large model in each test task, the performance characteristic index of the large model and the resource utilization rate of the large model in each test task are analyzed; the test response time of the large model in each test task is obtained; and the task generalization capability value of the large model is comprehensively determined, and the specific expression is:

[0010]

[0011] Where RW is the task generalization capability value of the large model, TZ is the performance characteristic index of the large model, and ZY j is the resource utilization rate of the large model in the jth test task, j is the number of each test task, j = 1, 2, 3, ..., n, n is the total number of test tasks, SC j is the test response time of the large model in each test task, a4 is the weight factor corresponding to the performance indicator predefined in the evaluation system control library, and a5 is the impact factor corresponding to the unit value of the test response time predefined in the evaluation system control library.

[0012] The second aspect of the present invention provides an evaluation method for the generalization ability of an AI large model, including: domain generalization ability value evaluation: based on the construction of real scenarios, obtaining a test generalization data set of the large model in each test domain, analyzing the domain generalization performance indicators of the large model in each test domain, and evaluating the domain generalization ability value of the large model; task generalization ability value evaluation: based on the construction of real scenarios, obtaining a test generalization data set of the large model in each test task, analyzing the task generalization performance indicators of the large model in each test task, and evaluating the task generalization ability value of the large model; generalization ability optimization: comparing the domain generalization ability value of the large model and the task generalization ability value of the large model with the preset domain generalization ability reference value and the task generalization ability reference value, respectively, to optimize the generalization ability of the large model.

[0013] A third aspect of the present invention provides an electronic device, characterized in that the electronic device comprises a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the system in an embodiment of the present invention.

[0014] Compared with the prior art, the embodiments of the present invention have at least the following advantages or beneficial effects:

[0015] (1) The present invention provides an evaluation system, method and electronic device for the generalization capability of an AI large model. First, the domain generalization capability value and the task generalization capability value of the large model are evaluated, and compared with the preset domain generalization capability reference value and task generalization capability reference value, respectively, so as to optimize the generalization capability of the large model, enable the large model to maintain good generalization performance, better adapt to changes in the actual application environment, improve the overall prediction accuracy of the large model, thereby enhancing the robustness of the large model, making the large model more stable and reliable in actual applications, and reducing erroneous evaluations caused by incomplete data analysis.

[0016] (2) The present invention obtains the domain generalization data set of the big model in each test field, analyzes the domain generalization performance index of the big model in each test field, and evaluates the domain generalization ability value of the big model. It can clarify the performance difference of the big model in different fields, so as to find out in which fields the big model has deficiencies, which helps to optimize the domain structure and parameters of the big model in a targeted manner and improve the performance of the big model in weak fields.

[0017] (3) The present invention obtains the task generalization data set of the big model in each test task, analyzes the task generalization performance index of the big model in each test task, and comprehensively evaluates the task generalization ability value of the big model. This can quantitatively reflect the performance of the big model on different tasks, which is helpful to comprehensively consider and evaluate the real task generalization ability of the big model; it is also helpful to discover the overfitting phenomenon of the big model in each test task, so as to take correct measures to reduce overfitting and improve the generalization reliability and stability of the big model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0019] Figure 1 It is a schematic diagram of system module connection of the present invention.

[0020] Figure 2 The figure is a schematic flow chart of the method steps of the present invention.

[0021] Figure 3 It is a performance characteristic curve diagram involved in the present invention.

[0022] As shown in the attached figure: 1. Horizontal reference line; 2. Vertical reference line. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0024] Reference Figure 1 As shown, the first aspect of the present invention provides an evaluation system for the generalization capability of a large AI model, including: a domain generalization capability value evaluation module, a task generalization capability value evaluation module, and a generalization capability optimization module.

[0025] The first aspect of the present invention provides an evaluation system for the generalization capability of a large AI model, which also includes: an evaluation system control library, wherein the evaluation system control library is used to store the defined rates of disk input and output, the domain generalization capability optimization plans corresponding to the generalization capability deviation value intervals of each domain, the task generalization capability optimization plans corresponding to the generalization capability deviation value intervals of each task, and the preset values ​​of various factors.

[0026] The domain generalization capability value evaluation module and the task generalization capability value evaluation module are both connected to the generalization capability optimization module, and the task generalization capability value evaluation module and the generalization capability optimization module are both connected to the evaluation system control library.

[0027] The domain generalization capability value evaluation module is used to obtain the domain generalization data set of the large model in each test domain based on the construction of real scenarios, analyze the domain generalization performance indicators of the large model in each test domain, and evaluate the domain generalization capability value of the large model.

[0028] It should be explained that the above-mentioned large model is a machine learning model with large-scale parameters and complex structure applied on a computer. When the large model runs on a computer to evaluate its generalization ability, the operating status of the required computer devices such as disks and central processing units will affect the evaluation of its generalization ability.

[0029] The above real scenarios include but are not limited to the medical field, the financial field, and intelligent manufacturing. Based on the construction of real scenarios, taking intelligent manufacturing as an example, the specific ones are:

[0030] First, it is necessary to clarify the specific application scenarios that the generalization of intelligent manufacturing models is to solve, such as production process optimization, quality control, predictive maintenance, etc.; collect relevant data from various intelligent manufacturing equipment and systems, including equipment status, production parameters, product quality information, etc.; select appropriate machine learning or deep learning models according to application scenarios and needs. For example, for production process optimization, you can choose a time series prediction model; for quality control, you can choose a classification or regression model; use the collected data to train the model. During the training process, you need to continuously adjust the model parameters and optimize the model performance; evaluate the generalization ability of the model through methods such as cross-validation to ensure that the model can maintain good generalization performance on new data.

[0031] Through the construction of the above-mentioned intelligent manufacturing, in the process of evaluating the generalization ability of the model, the domain generalization data set of the large model in each test field is obtained, wherein each test field may include production process optimization, quality control, predictive maintenance, etc.

[0032] Specifically, the domain generalization accuracy of the large model in each test domain, the specific analysis process is as follows:

[0033] Domain generalization datasets based on large models in various test domains, including prediction sample datasets of large models in various test domains.

[0034] According to the prediction sample data set of the large model in each test field, the number of true positive prediction samples of the large model in each test field, the number of true negative prediction samples of the large model in each test field, and the total number of prediction samples of the large model in each test field are extracted, wherein the number of samples involved in the embodiment of this aspect is statistically obtained in the prediction report of the large model, and the domain generalization accuracy Z of the large model in each test field is obtained by comprehensive analysis. i , where domain generalization accuracy refers to the proportion of samples that can be correctly predicted or classified when the large model is tested on multiple unseen and different data domains. The domain generalization accuracy in this embodiment can be obtained by the following relationship: domain generalization accuracy = {(the number of true positive prediction samples + the number of true negative prediction samples) / the total number of prediction samples}*100%. This embodiment only analyzes the domain generalization accuracy of a set of test domains.

[0035] It should be explained that the determination of the above-mentioned positive examples and negative examples depends on the actual prediction requirements of the forecast personnel for the large model in different test fields. Examples that meet the prediction requirements are recorded as positive examples, and examples that do not meet the prediction requirements are recorded as negative examples. This embodiment does not impose too many restrictions.

[0036] True positive prediction samples refer to samples that are actually positive examples in each test field and are predicted to be positive examples by the model; true negative prediction samples refer to samples that are actually negative examples in each test field and are predicted to be negative examples by the model; the total number of prediction samples refers to the number of true positive prediction samples, the number of true negative prediction samples, and the number of false positive prediction samples, that is, the number of samples that are actually negative examples in each test field but predicted to be positive examples by the model, and the number of false negative prediction samples, that is, the sum of the number of samples that are actually positive examples in each test field but predicted to be negative examples by the model.

[0037] Furthermore, the domain generalization recall rate of the large model in each test domain is analyzed in the following specific process:

[0038] According to the prediction sample data set of the big model in each test field, the number of false negative prediction samples of the big model in each test field is extracted, and the number of true positive prediction samples of the big model in each test field is combined to analyze and obtain the domain generalization recall rate ZH of the big model in each test field. i , where the domain generalization recall rate refers to the instances correctly identified as positive by the large model, that is, the proportion of true positive examples to all actual positive examples. The domain generalization recall rate in this embodiment can be obtained by the following relationship: domain generalization recall rate = {the number of true positive example prediction samples / (the number of true positive example prediction samples + the number of false negative example prediction samples)}*100%. This embodiment only analyzes the domain generalization recall rate of a set of test domains.

[0039] In this embodiment, the domain generalization recall rate of the above large model in each test domain is shown in Table 1:

[0040] Table 1 Numerical change relationship between domain generalization recall and its parameters

[0041] serial number True examples / False negative examples Domain generalization recall / percentage 1 300 200 60 2 150 180 45 3 200 150 57

[0042] In this embodiment, according to the table of the domain generalization recall rate of the large model in each test domain, when the number of true positive prediction samples decreases and the number of false negative prediction samples increases, the domain generalization recall rate of the large model in each test domain will decrease, which means that the large model cannot accurately identify samples that are actually positive examples. At this time, the generalization performance of the large model is poor.

[0043] Specifically, the domain generalization capability value of the large model is analyzed as follows:

[0044] In this embodiment, according to the domain generalization data set of the large model in each test field, the domain generalization accuracy of the large model in each test field and the domain generalization recall rate of the large model in each test field are analyzed, and the data is processed with the preset domain generalization reference accuracy rate and domain generalization reference recall rate to determine the domain generalization ability value of the large model. In this embodiment, a more accurate calculation method is used to obtain it, and the specific expression is:

[0045]

[0046] Wherein LY is the domain generalization capability value of the large model. In this embodiment, it refers to the prediction and processing capability of the large model in new domains and new scenarios outside the training data set of the test domain. The larger the domain generalization capability value, the stronger its prediction and processing capability. When the model predicts the number of correct samples, the higher the domain generalization accuracy will be, which means that the large model also has a stronger ability to recognize negative examples, which may reduce the domain generalization recall rate. Therefore, in practical applications, it is often necessary to strike a balance between accuracy and recall rate to enhance the domain generalization capability of the large model.

[0047] Z i is the domain generalization accuracy of the large model in the i-th test domain, which refers to the proportion of the number of samples correctly predicted by the large model in each test domain to the total number of samples.

[0048] i is the number of each test area, i = 1, 2, 3, ..., m, and m is the total number of test areas.

[0049] Z' is the domain generalization reference accuracy, which refers to the standard value of domain generalization accuracy.

[0050] ZH i is the domain generalization recall rate of the large model in the i-th test domain, which refers to the proportion of samples that are actually positive examples in each test domain that are predicted as positive examples by the large model.

[0051] ZH' is the domain generalization reference recall rate, which refers to the standard value of the domain generalization recall rate.

[0052] a1 is the correction factor corresponding to the domain generalization capability value predefined in the evaluation system control library.

[0053] In a specific embodiment, a fitting curve corresponding to the domain generalization capability value is obtained by fitting the historical training times of the large model, the historical sample distribution density, and the network approximation error, and the fitting curve corresponding to the domain generalization capability value is brought into the fitting curve according to the real-time training times of the large model, the sample distribution density, and the network approximation error, so as to obtain the correction factor corresponding to the domain generalization capability value in this embodiment. At the same time, the value range in this embodiment is (0, 1).

[0054] The task generalization capability value evaluation module is used to obtain the task generalization data set of the large model in each test task based on the construction of real scenarios, analyze the task generalization performance indicators of the large model in each test task, and comprehensively evaluate the task generalization capability value of the large model.

[0055] Furthermore, the performance characteristic index of the large model is specifically analyzed as follows:

[0056] According to the task generalization data set of the large model in each test task, we extract the number of true positive prediction samples of the large model in each test task, the number of false negative prediction samples of the large model in each test task, the number of true negative prediction samples of the large model in each test task, and the number of false positive prediction samples of the large model in each test task.

[0057] The above-mentioned testing tasks, taking intelligent manufacturing as an example, include but are not limited to functional testing, performance testing, and compatibility testing of intelligent manufacturing.

[0058] The number of true positive prediction samples of the large model in each test task and the number of false negative prediction samples of the large model in each test task are comprehensively processed to obtain the recall rate of the large model in each test task, which is recorded as the true positive rate of the large model in each test task. The relationship introduced by the comprehensive processing is the same as the relationship introduced by the domain generalization recall rate of the large model in each test field.

[0059] The number of false positive prediction samples of the large model in each test task and the number of true negative prediction samples of the large model in each test task are comprehensively processed to obtain the false positive rate of the large model in each test task. The false positive rate in this embodiment can be obtained by the following relationship: False positive rate = {number of false positive prediction samples / (number of false positive prediction samples + number of true negative prediction samples)}*100%. This embodiment only analyzes the false positive rate of a group of test tasks.

[0060] According to the true positive rate of the large model in each test task and the false positive rate of the large model in each test task, a performance characteristic curve of the large model is constructed. The area value under the performance characteristic curve is obtained through data processing, which is recorded as the performance characteristic index TZ of the large model.

[0061] Construct performance characteristic curves of large models, such as Figure 3 As shown, the horizontal axis of the performance characteristic curve is the false positive rate, the unit is percentage, and the vertical axis is the true positive rate, the unit is percentage.

[0062] According to the performance characteristic curve, the horizontal reference line 1 and the vertical reference line 2 are located therefrom, and the area value formed by the performance characteristic curve, the horizontal reference line 1 and the vertical reference line 2 is extracted. In this embodiment, the area value can be obtained by integrating the performance characteristic curve and recorded as the performance characteristic index of the large model. The specific integral relationship is:

[0063]

[0064] Where TZ is the performance characteristic index of the large model, X j is the false positive rate of the large model in the jth test task, X j+1 is the false positive rate of the large model in the j+1th test task, Y j is the true probability of the large model in the jth test task, Y j+1 is the true positive rate of the large model in the j+1th test task.

[0065] Specifically, the resource utilization of the large model in each test task is analyzed as follows:

[0066] According to the task generalization data set of the big model in each test task, the data input rate and data output rate of the disk belonging to the big model in the prediction period of each test task are extracted, and the average data input and output rate of the disk belonging to the big model in the prediction period of each test task is obtained through data processing, wherein the prediction period is determined by the evaluator based on factors such as the test task, task requirements, and prediction standards; the data input rate and the data output rate can be extracted from the usage report of the task manager carried by the evaluation system itself, and the data processing is specifically to average the data input rate and data output rate of the disk belonging to the big model in the prediction period of each test task to obtain the average data input and output rate of the disk belonging to the big model in the prediction period of each test task.

[0067] The above-mentioned disk is a storage device for storing data and is the main storage medium of the computer. When conducting functional testing, performance testing, and compatibility testing of intelligent manufacturing, a large amount of binary data is stored on the disk to ensure that the large model can process a large amount of data faster when processing test tasks in intelligent manufacturing scenarios, thereby improving the efficiency and accuracy of the large model in processing test task data.

[0068] Data extracted from the evaluation system control library defines the input and output rates.

[0069] The data stored in the above-mentioned evaluation system control library is obtained by fitting based on the task generalization data set in multiple statistical test tasks. For example, the defined rate of disk input and output is obtained by obtaining the rates of multiple disk input and output, and the evaluator determines that the disk input and output rates that meet the test tasks are averaged and used as the defined rate of disk input and output. In this way, the value in the evaluation system control library is obtained, and the value may vary depending on the test tasks, task requirements, and prediction standards. This embodiment does not make any special limitations on this.

[0070] Obtain the operating temperature of the central processing unit of the large model at each predicted time point in each test task, where the predicted time points are obtained by dividing the prediction period into each prediction time point, and the prediction time points are arranged in order from large to small. The operating temperature ranked first is recorded as the maximum temperature of the central processing unit of the large model within the prediction period in each test task.

[0071] The above-mentioned central processing unit is the core component of the computer system in the intelligent manufacturing scenario, responsible for executing program instructions and processing data; it is the key to the computer's execution of various complex testing tasks; by monitoring the utilization rate of the central processing unit, the large model can promptly identify the generalization performance bottleneck of the computer system in processing complex testing tasks, and take effective measures to improve the overall generalization performance of the large model running on the computer.

[0072] Comprehensively analyze the resource utilization of the large model in each test task. In this embodiment, it is obtained by comprehensive analysis of the average data input and output rate of the disk to which the large model belongs in the prediction period of each test task and the maximum temperature of the central processor of the large model in each test task in the prediction period. This is used to determine the resource utilization of the large model in each test task. In this embodiment, a more accurate calculation method is used to obtain it. The specific expression is:

[0073]

[0074] Where ZY jis the resource utilization rate of the large model in the jth test task. In this embodiment, it refers to the degree of use of various resources of the computer system by the large model in the computer when executing different test tasks. When the resource utilization rate is high, it means that the large model can be trained on more data, thereby absorbing more diverse and extensive knowledge, which helps to improve the prediction and processing capabilities of the large model in new data, new scenarios, and new fields outside the training data set, that is, the generalization capability. When the data input and output rate in the test task is high, it means that the execution efficiency of the large model is high, and the resource utilization rate in the computer system will increase accordingly. When the operating temperature of the central processing unit is high, it will affect the resource utilization rate in the computer system. When the temperature of the central processing unit is too high, it will cause the performance of the computer system to decline, which will affect the data input and output speed, resulting in a decrease in resource utilization in the computer system.

[0075] j is the number of each test task, j=1,2,3,...,n, and n is the total number of test tasks.

[0076] SL j is the average data input and output rate of the disk belonging to the large model during the prediction period of the jth test task, which refers to the rate at which the disk in the test task can complete data read and write operations during the prediction period.

[0077] SL' is the data input / output rate, which refers to the minimum value of the data input / output rate.

[0078] SY j is the maximum operating temperature of the CPU in the jth test task of the large model during the prediction period, which refers to the maximum temperature generated by the CPU in the test task during the prediction period.

[0079] a2 is the correction factor corresponding to the resource utilization predefined in the evaluation system control library.

[0080] In a specific embodiment, the fitting curve corresponding to the resource utilization is obtained by fitting the historical resource capacity, historical response time, and historical resource usage of the central processing unit in the test task, and the fitting curve corresponding to the resource utilization is brought into the fitting curve according to the real-time resource capacity, response time, and resource usage of the central processing unit, thereby obtaining the correction factor corresponding to the resource utilization in this embodiment, and the value range in this embodiment is (0, 1).

[0081] a3 is the impact factor corresponding to the operating temperature unit value predefined in the evaluation system control library.

[0082] In a specific embodiment, the fitting curve corresponding to the operating temperature is obtained by fitting the historical frequency, historical number of cores, and historical number of concurrent tasks of the central processing unit in the test task, and the fitting curve corresponding to the operating temperature is brought into the fitting curve according to the real-time frequency, number of cores, and number of concurrent tasks of the central processing unit, thereby obtaining the influence factor corresponding to the unit value of the operating temperature in this embodiment, and the value range in this embodiment is (0, 1).

[0083] In this embodiment, the data definition input and output rate is set to 500 megabytes per second, the correction factor corresponding to the resource utilization is set to 0.52, and the impact factor corresponding to the operating temperature unit value is set to 0.33.

[0084] In this embodiment, the resource utilization rate of the above large model in each test task is shown in Table 2:

[0085] Table 2 Relationship between resource utilization and its parameters

[0086]

[0087]

[0088] In this embodiment, according to the table of resource utilization of the large model in each test task, it can be seen that the average data input and output rate has a greater impact on the resource utilization of the large model in each test task. When the average data input and output rate is greater than the rate limit value, the resource utilization at this time will also be greater. Therefore, if you want to improve the resource utilization of the large model in each test task, you need to take corresponding measures to optimize the data input and output rate of the disk.

[0089] Furthermore, the task generalization capability value of the large model is analyzed in the following specific process:

[0090] In this embodiment, according to the task generalization data set of the large model in each test task, the performance characteristic index of the large model in each test task and the resource utilization rate of the large model in each test task are analyzed, and the task generalization capability value of the large model is comprehensively determined by obtaining the test response time of the large model in each test task. In this embodiment, a more accurate calculation method is used to obtain it, and the specific expression is:

[0091]

[0092] Wherein RW is the task generalization capability value of the large model, which in this embodiment refers to the prediction and processing capability of the large model in new fields and new scenarios outside the training data set of the test task. The larger the task generalization capability value is, the stronger its prediction and processing capability is. The performance characteristic index of the large model in each test task and the resource utilization rate of the large model in each test task are expressed as the performance index of the large model in each test task. The larger the performance index is, the stronger the task generalization capability is. When the test response time in the test task is longer, it indicates that the prediction capability of the large model is reduced, thereby affecting the task generalization capability of the large model. If the large model performs well on the training data set, but due to its high computational complexity, it may lead to a longer response time when processing new data. In this case, the test response time of the large model may be affected by its generalization performance.

[0093] TZ is the performance characteristic index of the large model, which represents the area value under the performance characteristic curve obtained through data processing and is recorded as the performance characteristic index of the large model.

[0094] ZY j is the resource utilization of the large model in the jth test task, which is obtained by comprehensive analysis of the disk input and output rate of the large model in each test task and the utilization rate of the central processing unit of the large model in each test task during the prediction period.

[0095] j is the number of each test task, j=1,2,3,...,n, and n is the total number of test tasks.

[0096] SC j It is the test response time of the big model in each test task, which refers to the time taken from receiving input data to generating prediction results when the big model makes predictions for different test tasks.

[0097] a4 is the weight factor corresponding to the predefined performance index in the evaluation system control library.

[0098] In a specific embodiment, the fitting curve corresponding to the performance indicator is obtained by fitting the historical performance characteristic index and the historical resource utilization in the test task, and the fitting curve corresponding to the performance indicator is introduced into the fitting curve according to the real-time performance characteristic index and resource utilization, thereby obtaining the weight factor corresponding to the performance indicator in this embodiment, and the value range in this embodiment is (0, 1).

[0099] a5 is the impact factor corresponding to the unit value of the test response time predefined in the evaluation system control library.

[0100] In a specific embodiment, the fitting curve corresponding to the test response time is obtained by fitting the historical parameter number, historical task complexity, and historical data size of the large model, and the fitting curve corresponding to the test response time is brought into the fitting curve according to the parameter number, task complexity, and data size of the real-time large model, thereby obtaining the impact factor corresponding to the unit value of the test response time in this embodiment, and the value range in this embodiment is (0, 1).

[0101] The generalization capability optimization module is used to compare the domain generalization capability value of the large model and the task generalization capability value of the large model with the preset domain generalization capability reference value and task generalization capability reference value, respectively, so as to optimize the generalization capability of the large model.

[0102] Specifically, the optimization of the generalization ability of the large model is achieved, and the specific analysis process is as follows:

[0103] The domain generalization ability value of the large model is compared with the preset domain generalization ability reference value. If the domain generalization ability value of the large model is greater than or equal to the domain generalization ability reference value, it means that the large model has shown high accuracy and stability in cross-domain applications, which means that the large model can not only handle the domain capabilities in the training data set well, but also maintain a high performance level in unseen domains. In this case, there is no need to optimize the domain generalization ability of the large model.

[0104] If the domain generalization capability value of the large model is less than the domain generalization capability reference value, the domain generalization capability value of the large model is differenced with the domain generalization capability reference value to obtain the domain generalization capability deviation value of the large model, and the domain generalization capability optimization plan corresponding to each domain generalization capability deviation value interval defined in the evaluation system control library is matched. The specific matching process is: first determine the interval to which the domain generalization capability deviation value of the large model belongs, and assign the domain generalization capability optimization plan corresponding to the interval to the large model corresponding to the domain generalization capability deviation value, so as to obtain the domain generalization capability optimization plan of the large model, thereby optimizing the domain generalization capability of the large model.

[0105] The above-mentioned domain generalization capability optimization plan is specifically as follows: expand the source and scope of domain generalization data sets, use data enhancement technology to increase the diversity and quantity of domain generalization data; adjust the complexity and number of parameters of the model to avoid overfitting or underfitting; and improve the domain generalization capability of the overall model by combining multiple models.

[0106] The task generalization ability value of the large model is compared with the preset task generalization ability reference value. If the task generalization ability value of the large model is greater than or equal to the task generalization ability reference value, it means that the large model has shown high accuracy and stability in cross-task applications, which means that the large model can not only handle the tasks in the training data set well, but also maintain a high performance level in unseen tasks. Therefore, there is no need to optimize the task generalization ability of the large model.

[0107] If the task generalization capability value of the large model is less than the task generalization capability reference value, the task generalization capability value of the large model is differenced with the task generalization capability reference value to obtain the task generalization capability deviation value of the large model, and matched with the task generalization capability optimization plan corresponding to each task generalization capability deviation value interval defined in the evaluation system control library. The specific matching process is: first determine the interval to which the task generalization capability deviation value of the large model belongs, and assign the task generalization capability optimization plan corresponding to the interval to the large model corresponding to the task generalization capability deviation value, so as to obtain the task generalization capability optimization plan of the large model, thereby optimizing the task generalization capability of the large model, and finally achieving the optimization of the generalization capability of the large model.

[0108] The above-mentioned task generalization ability optimization plan is specifically as follows: try to use the adaptive learning rate optimizer to control the update direction and amplitude of the model parameters to adjust the learning rate and improve the model training efficiency; use the model that has been trained on related tasks for transfer learning to accelerate the training process on new tasks and improve generalization ability; regularly evaluate the model's task generalization ability on unknown task data, and optimize and adjust the model based on the evaluation results.

[0109] Reference Figure 2 As shown, the second aspect of the present invention provides an evaluation method for the generalization ability of an AI large model, including: domain generalization ability value evaluation: based on the construction of real scenarios, obtaining a test generalization data set of the large model in each test domain, analyzing the domain generalization performance index of the large model in each test domain, and evaluating the domain generalization ability value of the large model; task generalization ability value evaluation: based on the construction of real scenarios, obtaining a test generalization data set of the large model in each test task, analyzing the task generalization performance index of the large model in each test task, and evaluating the task generalization ability value of the large model; generalization ability optimization: comparing the domain generalization ability value of the large model and the task generalization ability value of the large model with the preset domain generalization ability reference value and the task generalization ability reference value, respectively, to optimize the generalization ability of the large model.

[0110] A third aspect of the present invention provides an electronic device, characterized in that the electronic device comprises a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the system in an embodiment of the present invention.

[0111] The above contents are merely examples and explanations of the structure of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the structure of the invention or exceed the scope defined by the present invention, they should all fall within the protection scope of the present invention.

Claims

1. An evaluation system for the generalization ability of a large AI model, characterized in that: include: The domain generalization capability value evaluation module is used to obtain the domain generalization data set of the big model in each test domain based on the construction of real scenarios, analyze the domain generalization performance indicators of the big model in each test domain, and evaluate the domain generalization capability value of the big model; The task generalization capability value evaluation module is used to obtain the task generalization data set of the large model in each test task based on the construction of real scenarios, analyze the task generalization performance indicators of the large model in each test task, and comprehensively evaluate the task generalization capability value of the large model; The generalization capability optimization module is used to compare the domain generalization capability value of the large model and the task generalization capability value of the large model with the preset domain generalization capability reference value and task generalization capability reference value, so as to optimize the generalization capability of the large model.

2. According to claim 1, a system for evaluating the generalization ability of a large AI model is characterized by: The optimization of the generalization ability of the large model is achieved, and the specific analysis process is as follows: Compare the domain generalization capability value of the large model with the preset domain generalization capability reference value. If the domain generalization capability value of the large model is greater than or equal to the domain generalization capability reference value, there is no need to optimize the domain generalization capability of the large model. If the domain generalization capability value of the large model is less than the domain generalization capability reference value, perform difference processing on the domain generalization capability value of the large model and the domain generalization capability reference value to obtain the domain generalization capability deviation value of the large model, and match it with the domain generalization capability optimization plan corresponding to each domain generalization capability deviation value interval defined in the evaluation system control library to obtain the domain generalization capability optimization plan of the large model, so as to optimize the domain generalization capability of the large model. The task generalization capability value of the large model is compared with the preset task generalization capability reference value. If the task generalization capability value of the large model is greater than or equal to the task generalization capability reference value, there is no need to optimize the task generalization capability of the large model. If the task generalization capability value of the large model is less than the task generalization capability reference value, the task generalization capability value of the large model and the task generalization capability reference value are differenced to obtain the task generalization capability deviation value of the large model, which is matched with the task generalization capability optimization plan corresponding to each task generalization capability deviation value interval defined in the evaluation system control library to obtain the task generalization capability optimization plan of the large model, so as to optimize the task generalization capability of the large model and finally achieve the optimization of the generalization capability of the large model.

3. According to claim 1, a system for evaluating the generalization ability of a large AI model is characterized by: The domain generalization capability value of the large model is analyzed as follows: Based on the domain generalization data set of the big model in each test domain, the domain generalization accuracy of the big model in each test domain and the domain generalization recall of the big model in each test domain are analyzed. By processing the data with the preset domain generalization reference accuracy and domain generalization reference recall, the domain generalization ability value of the big model is comprehensively determined.

4. According to claim 1, a system for evaluating the generalization ability of a large AI model is characterized by: The task generalization capability value of the large model is analyzed in the following specific process: Based on the task generalization data set of the large model in each test task, the performance characteristic index of the large model and the resource utilization rate of the large model in each test task are analyzed; Get the test response time of the large model in each test task; The task generalization capability value of the large model is comprehensively determined, and the specific expression is: Where RW is the task generalization capability value of the large model, TZ is the performance characteristic index of the large model, and ZY j is the resource utilization rate of the large model in the jth test task, j is the number of each test task, j = 1, 2, 3, ..., n, n is the total number of test tasks, SC j is the test response time of the large model in each test task, a4 is the weight factor corresponding to the performance indicator predefined in the evaluation system control library, and a5 is the impact factor corresponding to the unit value of the test response time predefined in the evaluation system control library.

5. According to claim 3, a system for evaluating the generalization ability of a large AI model is characterized by: The domain generalization accuracy of the large model in each test domain, the specific analysis process is as follows: Domain generalization datasets based on the big model in each test domain, including prediction sample datasets of the big model in each test domain; According to the prediction sample data set of the big model in each test field, we extracted the number of true positive prediction samples of the big model in each test field, the number of true negative prediction samples of the big model in each test field, and the total number of prediction samples of the big model in each test field, and comprehensively analyzed to obtain the domain generalization accuracy of the big model in each test field.

6. According to claim 3, a system for evaluating the generalization ability of a large AI model is characterized by: The domain generalization recall rate of the large model in each test domain, the specific analysis process is as follows: According to the prediction sample data set of the big model in each test field, the number of false negative prediction samples of the big model in each test field is extracted, and the number of true positive prediction samples of the big model in each test field is combined to analyze and obtain the domain generalization recall rate of the big model in each test field.

7. The system for evaluating the generalization capability of a large AI model according to claim 4, characterized in that: The performance characteristic index of the large model, the specific analysis process is: According to the task generalization data set of the large model in each test task, the number of true positive prediction samples of the large model in each test task, the number of false negative prediction samples of the large model in each test task, the number of true negative prediction samples of the large model in each test task, and the number of false positive prediction samples of the large model in each test task are extracted; The number of true positive prediction samples of the large model in each test task and the number of false negative prediction samples of the large model in each test task are processed comprehensively to obtain the recall rate of the large model in each test task, which is recorded as the true positive rate of the large model in each test task; The number of false positive prediction samples of the large model in each test task and the number of true negative prediction samples of the large model in each test task are processed comprehensively to obtain the false positive rate of the large model in each test task; According to the true positive rate of the large model in each test task and the false positive rate of the large model in each test task, a performance characteristic curve of the large model is constructed. The area value under the performance characteristic curve is obtained through data processing and recorded as the performance characteristic index of the large model.

8. According to claim 4, a system for evaluating the generalization ability of a large AI model is characterized by: The resource utilization of the large model in each test task, the specific analysis process is as follows: According to the task generalization data set of the large model in each test task, the data input rate and data output rate of the disk to which the large model belongs in the prediction period of each test task are extracted, and the average data input and output rate of the disk to which the large model belongs in the prediction period of each test task is obtained through data processing; Extract data from the evaluation system control library to define input and output rates; The operating temperature of the central processing unit of the large model at each predicted time point in each test task is obtained, and they are arranged in order from large to small. The operating temperature ranked first is recorded as the maximum operating temperature of the central processing unit of the large model within the predicted period in each test task, thereby comprehensively analyzing the resource utilization of the large model in each test task.

9. A method for evaluating the generalization ability of a large AI model, characterized in that: include: Domain generalization capability value evaluation: Based on the construction of real scenarios, obtain the test generalization data set of the big model in each test domain, analyze the domain generalization performance indicators of the big model in each test domain, and evaluate the domain generalization capability value of the big model; Task generalization capability value evaluation: Based on the construction of real scenarios, obtain the test generalization data set of the large model in each test task, analyze the task generalization performance indicators of the large model in each test task, and evaluate the task generalization capability value of the large model; Generalization capability optimization: The domain generalization capability value of the large model and the task generalization capability value of the large model are compared with the preset domain generalization capability reference value and task generalization capability reference value respectively, so as to optimize the generalization capability of the large model.

10. An electronic device, characterized in that: The electronic device comprises a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the system described in any one of claims 1-8.

Images