Test feedback improvement method and system for improving performance of large model

Abstract

The invention discloses a test feedback improvement method and system for improving the performance of a large model, and relates to the technical field of computers. The method comprises the following steps: constructing a test set containing various types and difficulty tasks, and collecting and preprocessing multi-modal data; inputting the test set into the model, obtaining a generation result, extracting feature information and recording performance indexes; according to preset evaluation feature information, establishing a multi-dimensional evaluation index system for classification and weight distribution, and calculating a comprehensive score of the large model; analyzing advantages and deficiencies of the large model based on a quantitative evaluation result, and proposing targeted improvement suggestions; improving the large model according to improvement suggestions; and retesting the improved model by using the same test set, and comparing performance changes before and after improvement. The method can comprehensively evaluate the advantages and deficiencies of large model generation capability and accurate positioning, provides improvement suggestions and continuous optimization, improves the quality and diversity of generated results, and has a wide application prospect.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a test feedback improvement method and system for improving the performance of large models. Background Technology

[0002] In fields such as business intelligence and data analytics, large-scale models can analyze and mine vast amounts of textual and business data to extract valuable information and knowledge, supporting business decision-making. For example, by analyzing textual data such as market research reports and user reviews, companies can understand market demands and user feedback, enabling them to formulate marketing strategies and product development directions.

[0003] However, existing technologies lack a systematic and comprehensive approach to evaluating the ability to generate large models. The test data used has limited coverage and cannot fully reflect the true performance of the model under different types of tasks and difficulty levels. They only focus on the model's performance on specific types of tasks or single performance indicators, resulting in general and untargeted improvement measures for large models, leading to unsatisfactory improvement results. At the same time, there is a lack of effective feedback mechanisms to continuously track changes in model performance and make corresponding adjustments, making it difficult to achieve gradual improvement and continuous optimization of large model performance and easily falling into the dilemma of optimization stagnation. In order to address the shortcomings of existing technologies, this invention provides a test feedback improvement method and system for improving the performance of large models, so as to solve the above problems. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a test feedback improvement method and system for enhancing the performance of large models. By constructing a test set containing various types and difficulties of tasks and collecting multimodal data, the test set is made widely representative. Combining quantitative evaluation with a multi-dimensional indicator system, the performance and strengths and weaknesses of large models are accurately determined. Based on precise analysis, improvement suggestions covering multiple key aspects are proposed, forming a feedback loop that repeats the analysis, improvement, and testing steps until the model performance reaches the predetermined target, thus achieving continuous optimization.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a test feedback improvement method for improving the performance of large models, characterized by comprising the following steps: Step S1: Construct a test set that covers multiple types of tasks for determining large model generation, and collect raw data covering different domains for each type of task and preprocess it; Step S2: Input the test set into the model to obtain the generated results, extract the feature information of the generated results and record the performance indicators; Step S3: Based on the preset evaluation feature information, establish a multi-dimensional evaluation index system for classification and weight allocation, and calculate the comprehensive score A of the large model according to the formula [Comprehensive score A = Σ (Indicator score Xi × Corresponding indicator weight Wi)]. Step S4: Based on the quantitative evaluation results, analyze the advantages and disadvantages of the large model, compare the comprehensive scores of each test task type with the preset performance threshold, and determine the good and bad parts of the model performance; Step S5: Based on the identified shortcomings, propose targeted improvements to the model design; Step S6: Improve the large model according to the improvement suggestions; Step S7: Retest the improved large model using the same test set, following the same procedure as in step S2, to compare the performance changes of the model before and after the improvement; Step S8: Compare the performance of the large model before and after the improvement to quantitatively evaluate the improvement effect, compare the increase in the comprehensive score A, and use statistical methods to test the significance of the improvement effect; Step S9: If the model performance does not meet the predetermined target requirements, repeat steps S4 to S8 until the quality and diversity of the large model generation results meet the predetermined target, forming a continuous optimization closed-loop feedback mechanism.

[0006] Preferably, in step S1, data augmentation techniques are used to augment the original data when constructing the test set.

[0007] Preferably, in step S2, the extracted feature information of the generated result includes the semantic coherence, logic, and diversity indicators of the generated text, the clarity and detail reproduction of the generated image, and the fluency and sound quality indicators of the generated speech.

[0008] Preferably, in step S3, when establishing a multi-dimensional evaluation index system, the weight coefficients of each evaluation index are determined according to the characteristics of the generated task and the actual application scenario requirements. The weight coefficients are set in the range of 0.1 to 0.5, and the sum of all weight coefficients is 1. A quantitative standard range is set for the evaluation indexes. The semantic coherence index score range is 0-100, below 60 points is considered incoherent, 60-80 points is considered basically coherent, and above 80 points is considered highly coherent.

[0009] Preferably, in step S4, data visualization technology is used to present the evaluation results intuitively, a bar chart is generated to show the comparison of comprehensive scores under different task types, a line chart is drawn to show the score change curve of the model under different difficulty levels of tasks, and a heat map is created to show the distribution of the characteristic indicators of the model generation results in different dimensions.

[0010] Preferably, in step S5, the proposed improvement suggestions for model design also include the optimization and adjustment of model parameters, including proposing a dynamic adjustment strategy for the learning rate parameter during model training; and proposing optimization suggestions for the regularization parameter in the model based on whether the model is overfitting or underfitting.

[0011] Preferably, in step S6, a distributed training architecture is used to improve the large model by dividing it into multiple sub-models, which are then distributed to different computing nodes for parallel training. Communication strategies are optimized to reduce communication overhead between nodes, and the training status and resource usage of each computing node are monitored.

[0012] Preferably, in step S8, when using statistical methods to test the significance of the evaluation results, a paired t-test or analysis of variance is selected, the significance level α is set to 0.05, the null hypothesis is rejected based on the test statistic, and the effect size is calculated to measure the actual significance of the improvement effect. The effect size calculation formula is: effect size d = (difference between mean before and after improvement) / standard deviation.

[0013] Preferably, in step S9, after determining that the performance of the large model meets the predetermined target requirements, the improved model is subjected to verification testing before actual application deployment. This includes collecting actual application data in real business scenarios to construct a verification test set, inputting the verification test set into the improved model and testing it according to the actual application process, comparing the verification test results with the predetermined actual application performance index requirements, stipulating that the model's accuracy in the actual scenario reaches more than 95% and the response time does not exceed 1 second. If the requirements are not met, the model is readjusted, the system configuration is optimized, and verification testing is conducted again.

[0014] A second aspect of the present invention provides a test feedback improvement system for improving the performance of large models, the system comprising: The test set module is used to build test sets that cover various types of tasks for determining large model generation. The result generation module is used to input the test set into the model to obtain the generated results, extract the feature information of the generated results, and record the performance indicators. The scoring calculation module is used to establish a multi-dimensional evaluation index system based on preset evaluation feature information, classify and assign weights, and calculate the comprehensive score A of the large model according to the formula [Comprehensive Score A = Σ (Indicator Score Xi × Corresponding Indicator Weight Wi)]. The model evaluation module is used to analyze the advantages and disadvantages of the large model based on the quantitative evaluation results, compare the comprehensive score of each test task type with the preset performance threshold, and determine the good and bad parts of the model performance. The improvement suggestion module is used to propose targeted improvement suggestions for the model design based on the shortcomings identified in the analysis. The model improvement module is used to improve large models based on improvement suggestions. The retest module is used to retest the improved large model with the same test set, in the same way as step S2, to compare the performance changes of the model before and after the improvement. The improvement effect evaluation module is used to compare the performance of the large model before and after the improvement to quantitatively evaluate the improvement effect, compare the improvement of the comprehensive score A, and use statistical methods to test the significance of the improvement effect.

[0015] This invention discloses a test feedback improvement method and system for improving the performance of large models, which has the following beneficial effects: 1. This improved test feedback method for enhancing the performance of large models, by constructing a test set covering various task types and difficulty levels, enables a comprehensive and detailed evaluation of the large model's generation capabilities. Collecting multimodal data from different domains ensures the test set has broad representativeness, fully reflecting the large model's performance in various practical application scenarios. The quantitative evaluation process, combined with a multi-dimensional evaluation index system and weight allocation, accurately determines the large model's scores and overall performance across different task types and difficulty levels, thereby precisely identifying the model's strengths and weaknesses.

[0016] 2. This test feedback improvement method for improving the performance of large models, based on precise analysis of the shortcomings of large models, can propose targeted improvement suggestions, covering multiple key aspects such as model architecture, training data, and training methods. It provides clear directions and specific measures for model optimization, forming a feedback loop mechanism. By repeatedly executing steps such as analysis, improvement, and testing, the performance of large models is continuously optimized until the predetermined target is reached, thereby continuously improving the quality and diversity of generated results.

[0017] 3. This test feedback improvement method for enhancing the performance of large models utilizes data visualization techniques to intuitively present complex evaluation results, such as generating bar charts, line graphs, and heatmaps. This allows researchers to quickly and clearly understand the model's performance and identify problems, facilitating efficient decision-making and improvement. The method of this invention is applicable to different types of large models and various generation tasks, demonstrating strong adaptability. Whether for text generation, image generation, or speech generation tasks, performance evaluation and improvement can be achieved by constructing corresponding test sets and evaluation index systems, showing broad application prospects. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the overall method of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This application provides a test feedback improvement method and system for improving the performance of large models, which solves the problems of existing technologies lacking systematicness and comprehensiveness in evaluating the ability to generate large models, having limited coverage of test data, failing to fully reflect the true performance of models under different types of tasks and different difficulty levels, focusing only on the performance of models on specific types of tasks or single performance indicators, and having relatively general improvement measures for large models.

[0022] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0023] Example 1: This invention discloses a test feedback improvement method for improving the performance of large models, according to the appendix. Figure 1 As shown, it includes the following steps: Step S1: Construct the test set, which specifically includes: S1.1: Determine the various types of large model generation tasks, such as text generation, image generation, speech generation, etc. S1.2: For each type of task, collect raw data covering different fields. For example, for text generation tasks, collect text data from different fields such as news, novels, and academic papers. S1.3: Preprocess the collected raw data, including cleaning, labeling, and classification, to ensure data quality and usability; S1.4: Based on the difficulty requirements of the task, the preprocessed data is classified into different levels to form test subsets of different difficulty levels. For example, the text generation task is divided into different difficulty levels such as simple description generation, complex plot generation, and professional domain knowledge generation. S1.5: Integrate the test subsets of each difficulty level into a complete test set, and set clear evaluation indicators and expected outputs for each test item.

[0024] Step S2: Test the large model using the constructed test set. The specific steps are as follows: S2.1: Input the test set into the large model, so that the model processes it according to its generation logic and outputs the generation results; S2.2: Record the generation process information of the model in real time when processing each test item, including but not limited to the intermediate layer output of the model, attention weight distribution, etc. S2.3: Extract features from the generated results. The extracted feature information includes the semantic coherence, logic, and diversity indicators of the generated text, and the clarity and detail reproduction of the generated image. S2.4: Record performance indicators such as generation speed and resource consumption of the generated results in order to evaluate the efficiency of the model in a real-world operating environment.

[0025] Step S3: Based on the preset evaluation criteria, quantitatively evaluate the acquired generation results and their feature information. The specific method is as follows: S3.1: Establish a multi-dimensional evaluation index system, classify and assign weights to different types of indicators. For example, set the weight of semantic coherence index to 0.3, logicality index to 0.3, diversity index to 0.2, generation speed index to 0.1, resource consumption index to 0.1, etc. The weight allocation can be adjusted according to the key needs of different types of generation tasks. S3.2: Use specific quantization algorithms to score each evaluation metric. For example, for semantic coherence, a pre-trained language model can be used to evaluate the fluency of the generated text and output a score of 0-100; for image sharpness, an image quality evaluation algorithm can be used to calculate a sharpness metric score. S3.3: Calculate the overall score of the large model under each test task type and difficulty level based on the weights and scores of each indicator. The overall score = Σ (indicator score × indicator weight). At the same time, calculate the overall performance score of the model. The overall score is obtained by weighted averaging of the scores of each task type or other statistical methods.

[0026] Step S4: Based on the quantitative evaluation results, analyze the strengths and weaknesses of the large model. Specific analysis methods include: S4.1: Compare the overall score of each test task type and difficulty level with the preset performance threshold to determine which tasks the model performs well (score above the threshold) and which tasks it is deficient in (score below the threshold). S4.2: Conduct in-depth analysis of the characteristics of the model's generated results on different tasks. For example, for text generation tasks, analyze the richness of vocabulary, the complexity of sentence structure, and the degree of deviation from the topic in the generated text to find the specific reasons for the high or low scores; for image generation tasks, analyze the color accuracy, composition rationality, and matching degree with the input prompts of the image. S4.3: Use data visualization techniques to present the evaluation results in a visual way in the form of charts, such as generating bar charts to show the score comparison of different task types, generating heat maps to show the performance change trend of the model when dealing with tasks of different difficulty levels, so that researchers can quickly and accurately locate the problem.

[0027] Step S5: Based on the identified shortcomings, propose targeted suggestions for improving the model design. Specific suggestions include: S5.1: Suggestions for adjusting the model architecture. For example, if it is found that the model has insufficient coherence when processing long text generation tasks, it is recommended to increase the number of attention mechanism layers in the model or adopt a more advanced long sequence processing architecture, such as extending the traditional Transformer architecture to the Longformer architecture. S5.2: Optimization strategies for training data, such as if the model performs poorly on generation tasks in a specific domain, it is recommended to collect more high-quality data in that domain and integrate it into the training dataset according to a certain data distribution and proportion; at the same time, methods for data augmentation can be proposed, such as synonym replacement and sentence restructuring for text data, and rotation, flipping, and cropping for image data, to increase the diversity and coverage of the data. S5.3: Improvements to training methods include adjusting the learning rate scheduling strategy during training, adopting more suitable optimization algorithms, such as switching from the Adam optimizer to the LAMB optimizer, and introducing regularization terms to prevent overfitting. In addition, multi-task joint training methods can be proposed to enable the model to learn different types of tasks simultaneously during training, thereby improving its generalization ability and overall performance.

[0028] Step S6: Following the proposed improvement suggestions, improve the large model. The specific improvement process is as follows: S6.1: Based on the model architecture adjustment suggestions, redesign and reconstruct the neural network structure of the large model, including adding or reducing network layers, modifying the connection between layers, adjusting the number of neurons, etc. During the adjustment process, adopt a modular design method to ensure the flexibility and scalability of the architecture modification. S6.2: In accordance with the training data optimization strategy, update and expand the training dataset, including a series of operations such as data collection, preprocessing, labeling, and filtering, to ensure the quality and effectiveness of the new dataset; at the same time, reasonably mix and organize the original data and the newly added data to form a new batch of training data. S6.3: Based on the training method improvement measures, adjust the parameter settings and algorithm selection during the training process, such as resetting the initial learning rate, decay rate and other parameters, and writing new optimizer code; during the training process, adopt a distributed training architecture, divide the model into multiple parts and train them in parallel on different computing nodes to speed up the training and improve the efficiency of improvement; at the same time, monitor the model loss value, accuracy and other indicators during the training process in real time, and adjust the training strategy in a timely manner according to the changes in the indicators.

[0029] Step S7: Retest the improved large model using the same test set. The specific process is the same as in step S2, including inputting the test set into the model, recording the generation process information, extracting the generation result features, and recording performance indicators to ensure the consistency of test conditions and methods, so as to accurately compare the changes in model performance before and after the improvement.

[0030] Step S8: Compare the performance of the large model before and after the improvement on each test item, and quantitatively evaluate the improvement effect, specifically including: S8.1: Compare the overall scores of the model before and after the improvement under various test task types and difficulty levels, and calculate the improvement rate (improvement rate = (score after improvement - score before improvement) / score before improvement × 100%) to intuitively reflect the improvement effect of the improvement suggestions on model performance; S8.2: Compare the changes in feature indicators of the model generation results before and after the improvement, such as whether the diversity indicator of text generation has been significantly improved, whether the clarity indicator of image generation has been significantly improved, etc., and evaluate the effectiveness of the improvement measures in different aspects through the changes in specific values; S8.3: Compare the performance indicators of the model before and after the improvement, such as whether the generation speed is faster or the resource consumption is reduced, to determine the impact of the improvement operation on the actual running efficiency of the model; at the same time, use statistical methods (such as t test, ANOVA, etc.) to test the significance of the evaluation results to ensure the reliability and objectivity of the improvement effect. Set the significance level (such as α=0.05). Only when the improved indicators are statistically significantly different from those before the improvement are the improvement measures considered effective.

[0031] Step S9: Based on the comparative evaluation results, if the model performance does not meet the predetermined target requirements (the predetermined target can be set according to the actual application scenario and performance requirements, such as requiring the overall comprehensive performance score of the model to reach above 85 points, or the score of a specific key task type to reach above 90 points, etc.), then repeat steps S4 to S8 until the quality and diversity of the generated results of the large model meet the predetermined target requirements; during the repeated execution, the improvement suggestions of each iteration should be analyzed and optimized in depth based on the previous improvement to avoid repeated and ineffective improvement operations, forming a closed-loop feedback mechanism for continuous optimization, and gradually improving the performance of the large model.

[0032] In step S1, when constructing the test set, data augmentation techniques are used to expand the original data, specifically including: S1.6: For text data, new text samples are generated by methods such as synonym replacement, sentence structure adjustment, and adding stylized descriptions. For example, "This is a cute cat" is replaced with "This is an adorable kitten". The expression of the text is enriched by changing the vocabulary and sentence structure. S1.7: For image data, image processing algorithms such as rotation, flipping, scaling, and adding noise are used to generate image samples with different changes. For example, an original landscape image is rotated 90 degrees and flipped horizontally to generate a new image sample. At the same time, an image generation model is combined to generate an image with similar features to the original image but different details to increase the diversity and coverage of image data. S1.8: For speech data, new speech samples are generated by processing methods such as speed variation, pitch variation, and adding background noise. For example, the speed of a speech segment is increased or decreased, the pitch is changed, and speech scenarios in different environments are simulated, so that the test set can more comprehensively evaluate the robustness and adaptability of the large model in speech generation tasks.

[0033] The feature information recorded in step S2 also includes a similarity index between the generated result and the expected output, which is calculated as follows: S2.4.1: For text generation tasks, text similarity algorithms (such as cosine similarity, edit distance, etc.) are used to calculate the similarity score between the generated text and the expected text. For example, the generated text and the expected text are vectorized and then their cosine similarity values ​​are calculated, ranging from 0 to 1. The higher the value, the higher the similarity. S2.4.2: For image generation tasks, use image feature matching algorithms (such as SIFT feature matching, CNN feature extraction and comparison, etc.) to calculate the similarity index between the generated image and the expected image. For example, extract the high-level semantic feature vector of the image through a pre-trained CNN model, and then calculate the Euclidean distance or cosine similarity between the feature vectors of the generated image and the expected image to quantify the degree of similarity between them. S2.4.3: For speech generation tasks, acoustic feature comparison methods (such as Mel-frequency cepstral coefficient (MFCC) comparison, fundamental frequency comparison, etc.) are used to calculate the similarity index between the generated speech and the expected speech. For example, the MFCC feature parameters of the generated speech and the expected speech are extracted, and their root mean square error (RMSE) or other statistical distance indexes are calculated to evaluate the similarity of the speech.

[0034] The evaluation criteria in step S3 adopt a multi-dimensional comprehensive evaluation system, which weights different types of indicators, specifically including: S3.1.1: Based on the characteristics of the generation task and the needs of the actual application scenario, determine the weight coefficients of each evaluation indicator. For example, for news text generation tasks, the weights of semantic coherence and logicality indicators can be appropriately increased, while the weights of diversity indicators can be appropriately decreased; for creative writing generation tasks, the weights of diversity indicators can be increased, while the strict requirements for semantic coherence can be reduced. S3.1.2: Establish quantitative standard ranges for evaluation indicators. For example, specify that the semantic coherence indicator score range is 0-100, below 60 points is incoherent, 60-80 points is basically coherent, and above 80 points is highly coherent; the generation speed indicator is measured in the number of tokens generated per second, and specify reasonable speed ranges for different task types, etc. S3.1.3: When calculating the comprehensive score, a normalization method is used to process the scores of each indicator to make them comparable and additive. For example, all indicator scores are uniformly mapped to the range of 0-1, and then weighted and summed according to their weights to obtain the final comprehensive score. The formula is: Comprehensive score = Σ (normalized indicator score × indicator weight). This ensures that indicators with different dimensions and ranges can be organically combined to accurately reflect the performance level of the large model.

[0035] In step S4, when analyzing the strengths and weaknesses of the large model, data visualization technology is used to present the evaluation results intuitively, specifically including: S4.3.1: Generate a bar chart to show the comparison of the overall scores under different task types. The horizontal axis represents the task type and the vertical axis represents the overall score. Different colored bars are used to distinguish the scores before and after the improvement, and to intuitively present the performance change trend of the model in each task type. S4.3.2: Draw a line graph to show the score change curves of the model under different difficulty levels of tasks. The horizontal axis represents the difficulty level and the vertical axis represents the score. By comparing the trend and height of the curves before and after the improvement, analyze the model's adaptability to tasks of different difficulty and the improvement effect. S4.3.3: Create a heatmap to show the distribution of feature indicators of the model's generated results across different dimensions. For example, use the richness of vocabulary, sentence length, and topic relevance of the generated text as dimensions, and use the intensity of color to represent the magnitude of the indicator values. Identify the model's strengths and weaknesses in feature performance, providing an intuitive basis for improvement suggestions.

[0036] The improvement suggestions for model design proposed in step S5 also include the optimization and adjustment of model parameters, specifically including: S5.4: For the learning rate parameter in the model training process, a dynamic adjustment strategy is proposed. For example, a larger learning rate is used in the early stage of training to achieve rapid convergence. As the training progresses, the learning rate is gradually reduced to finely adjust the model parameters. Cosine annealing learning rate scheduling algorithm or adaptive learning rate scheduling algorithm can be used to dynamically adjust the learning rate according to the change of training loss value. S5.5: For regularization parameters in the model, such as L1 and L2 regularization coefficients or dropout probabilities, propose optimization suggestions based on the overfitting or underfitting of the model on the test set. For example, if the model performs well on the training set but poorly on the test set (overfitting), increase the regularization coefficient or dropout probability appropriately; if the model performs poorly on both the training and test sets (underfitting), decrease the regularization coefficient or dropout probability appropriately to balance the model's complexity and generalization ability. S5.6: Perform grid search or Bayesian optimization on the hyperparameters of the model. For example, set the combination of hyperparameter values ​​within a certain range, evaluate the performance of the model under different combinations through multiple experiments, and select the optimal hyperparameter configuration. For example, optimize and adjust hyperparameters such as the hidden layer dimension, the number of attention heads, and the number of encoder layers in the transformer model.

[0037] When performing the improvement operation on the large model in step S6, a distributed training architecture is adopted, specifically including: S6.4: Divide the large model into multiple sub-models or model fragments and distribute them to different computing nodes (such as GPUs, TPUs, etc.) for parallel training. For example, using data parallelism, the training dataset is divided into multiple subsets and distributed to various computing nodes. Each node calculates the gradient of its subset, and then the gradients are aggregated and the model parameters are updated through a parameter server or All-Reduce algorithm. Alternatively, using model parallelism, different layers or modules of the model are distributed to different nodes, and data interaction during forward and backward propagation is performed through an efficient message passing interface (MPI). S6.5: In the distributed training process, optimize the communication strategy to reduce the communication overhead between nodes. For example, use gradient compression techniques (such as quantization compression, sparse compression, etc.) to reduce the amount of data transmitted, and use asynchronous communication mechanisms or hybrid parallel strategies (combining data parallelism and model parallelism) to improve training efficiency. S6.6: Monitors the training status and resource usage of each computing node to ensure the stability and efficiency of the distributed training process. By dynamically adjusting the task allocation and resource allocation of computing nodes, it improves the overall training performance and accelerates the model improvement speed.

[0038] In step S8, when comparing the performance of the large model before and after the improvement, statistical methods are used to perform significance tests on the evaluation results, specifically including: S8.3.1: Select an appropriate significance test method. For example, for paired samples (model performance indicators before and after improvement), paired t-tests can be used, and for multiple independent samples, analysis of variance (ANOVA) and other methods can be used. S8.3.2: Set the significance level (e.g., α=0.05 or α=0.01), and determine whether to reject the null hypothesis (i.e., there is no significant difference in model performance before and after the improvement) based on the calculation results of the test statistic. If the null hypothesis is rejected, it is considered that the improvement measures have a significant effect on the model performance; otherwise, it is considered that the improvement effect is not significant and further optimization of the improvement plan is needed. S8.3.3: Calculate the effect size (such as Cohen's d value) to measure the practical significance of the improvement effect. The effect size can be expressed as the ratio of the difference between the mean and the standard deviation before and after the improvement. Even if the improvement effect is statistically significant, it is still necessary to evaluate its practical application value through the effect size. For example, an effect size greater than 0.8 indicates that it has great practical significance, around 0.5 indicates moderate significance, and less than 0.2 indicates that it has little significance.

[0039] After determining that the performance of the large model meets the predetermined target requirements in step S9, the process also includes verification testing of the improved model before its practical application deployment, specifically including: S9.1: Collect real-world application data in real business scenarios and build a verification test set. This verification test set should cover as many situations and data distributions as possible that may be encountered in real-world applications, including edge cases and special scenarios. S9.2: Input the validation test set into the improved model, test it according to the actual application process, and record the actual performance indicators such as the model's response results, running efficiency, and resource consumption; S9.3: Compare and verify the test results with the predetermined actual application performance indicators (such as specifying that the model's accuracy in the actual scenario reaches more than 95% and the response time does not exceed 1 second, etc.) to evaluate whether the model can run stably and efficiently in the real environment; S9.4: If the verification test results do not meet the requirements of actual application, the model shall be readjusted or the system configuration shall be optimized according to the cause of the problem, and the verification test shall be carried out again until the model can meet the requirements of actual deployment application, so as to ensure the smooth transition and reliable operation of the model from the laboratory environment to the actual application environment.

[0040] A second aspect of the present invention provides a test feedback improvement system for improving the performance of large models, the system comprising: The test set module is used to build test sets that cover various types of tasks for determining large model generation. The result generation module is used to input the test set into the model to obtain the generated results, extract the feature information of the generated results, and record the performance indicators. The scoring calculation module is used to establish a multi-dimensional evaluation index system based on preset evaluation feature information, classify and assign weights, and calculate the comprehensive score A of the large model according to the formula [Comprehensive Score A = Σ (Indicator Score Xi × Corresponding Indicator Weight Wi)]. The model evaluation module is used to analyze the advantages and disadvantages of the large model based on the quantitative evaluation results, compare the comprehensive score of each test task type with the preset performance threshold, and determine the good and bad parts of the model performance. The improvement suggestion module is used to propose targeted improvement suggestions for the model design based on the shortcomings identified in the analysis. The model improvement module is used to improve large models based on improvement suggestions. The retest module is used to retest the improved large model with the same test set, in the same way as step S2, to compare the performance changes of the model before and after the improvement. The improvement effect evaluation module is used to compare the performance of the large model before and after the improvement to quantitatively evaluate the improvement effect, compare the improvement of the comprehensive score A, and use statistical methods to test the significance of the improvement effect.

[0041] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0042] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A test feedback improvement method for improving the performance of large models, characterized in that, Includes the following steps: Step S1: Construct a test set that covers multiple types of tasks for determining large model generation, and collect raw data covering different domains for each type of task and preprocess it; Step S2: Input the test set into the model to obtain the generated results, extract the feature information of the generated results and record the performance indicators; Step S3: Based on the preset evaluation feature information, establish a multi-dimensional evaluation index system for classification and weight allocation, and calculate the comprehensive score A of the large model according to the formula [Comprehensive score A = Σ (Indicator score Xi × Corresponding indicator weight Wi)]. Step S4: Based on the quantitative evaluation results, analyze the advantages and disadvantages of the large model, compare the comprehensive scores of each test task type with the preset performance threshold, and determine the good and bad parts of the model performance; Step S5: Based on the identified shortcomings, propose targeted improvements to the model design; Step S6: Improve the large model according to the improvement suggestions; Step S7: Retest the improved large model using the same test set, following the same procedure as in step S2, to compare the performance changes of the model before and after the improvement; Step S8: Compare the performance of the large model before and after the improvement to quantitatively evaluate the improvement effect, compare the increase in the comprehensive score A, and use statistical methods to test the significance of the improvement effect; Step S9: If the model performance does not meet the predetermined target requirements, repeat steps S4 to S8 until the quality and diversity of the large model generation results meet the predetermined target, forming a continuous optimization closed-loop feedback mechanism.

2. The test feedback improvement method for improving the performance of large models according to claim 1, characterized in that, In step S1, data augmentation techniques are used to augment the original data when constructing the test set.

3. The test feedback improvement method for improving the performance of large models according to claim 2, characterized in that, In step S2, the extracted feature information of the generated result includes the semantic coherence, logic, and diversity indicators of the generated text, the clarity and detail reproduction of the generated image, and the fluency and sound quality indicators of the generated speech.

4. The test feedback improvement method for improving the performance of large models according to claim 3, characterized in that, In step S3, when establishing a multi-dimensional evaluation index system, the weight coefficients of each evaluation index are determined according to the characteristics of the generated task and the actual application scenario requirements. The weight coefficients are set in the range of 0.1 to 0.5, and the sum of all weight coefficients is 1. A quantitative standard range is set for the evaluation indexes. The semantic coherence index score range is 0-100, below 60 points is considered incoherent, 60-80 points is considered basically coherent, and above 80 points is considered highly coherent.

5. The test feedback improvement method for improving the performance of large models according to claim 4, characterized in that, In step S4, data visualization technology is used to present the evaluation results intuitively. A bar chart is generated to show the comparison of comprehensive scores under different task types, a line chart is drawn to show the score change curves of the model under different difficulty levels, and a heat map is created to show the distribution of the characteristic indicators of the model's generated results in different dimensions.

6. The test feedback improvement method for improving the performance of large models according to claim 5, characterized in that, In step S5, the proposed improvements to the model design also include optimization and adjustment of model parameters, including a dynamic adjustment strategy for the learning rate parameter during model training; and optimization suggestions for the regularization parameter in the model based on whether the model is overfitting or underfitting.

7. The test feedback improvement method for improving the performance of large models according to claim 1, characterized in that, In step S6, a distributed training architecture is used to improve the large model by dividing it into multiple sub-models. The sub-models are assigned to different computing nodes for parallel training. The communication strategy is optimized to reduce the communication overhead between nodes and the training status and resource usage of each computing node are monitored.

8. The test feedback improvement method for improving the performance of large models according to claim 1, characterized in that, In step S8, when statistical methods are used to test the significance of the evaluation results, paired t-test or analysis of variance are selected, and the significance level α is set to 0.

05. The null hypothesis is rejected based on the test statistic, and the effect size is calculated to measure the actual significance of the improvement. The effect size calculation formula is: effect size d = (difference between mean before and after improvement) / standard deviation.

9. The test feedback improvement method for improving the performance of large models according to claim 1, characterized in that, In step S9, after confirming that the performance of the large model meets the predetermined target requirements, the improved model is subjected to verification testing before actual application deployment. This includes collecting actual application data in real business scenarios to build a verification test set, inputting the verification test set into the improved model and testing it according to the actual application process, comparing the verification test results with the predetermined actual application performance index requirements, stipulating that the model's accuracy in the actual scenario should reach more than 95% and the response time should not exceed 1 second. If the requirements are not met, the model is readjusted, the system configuration is optimized, and the verification test is conducted again.

10. A test feedback improvement system for improving the performance of large models, applied to a test feedback improvement method for improving the performance of large models as described in any one of claims 1-9, characterized in that, The system includes: The test set module is used to build test sets that cover various types of tasks for determining large model generation. The result generation module is used to input the test set into the model to obtain the generated results, extract the feature information of the generated results, and record the performance indicators. The scoring calculation module is used to establish a multi-dimensional evaluation index system based on preset evaluation feature information, classify and assign weights, and calculate the comprehensive score A of the large model according to the formula [Comprehensive Score A = Σ (Indicator Score Xi × Corresponding Indicator Weight Wi)]. The model evaluation module is used to analyze the advantages and disadvantages of the large model based on the quantitative evaluation results, compare the comprehensive score of each test task type with the preset performance threshold, and determine the good and bad parts of the model performance. The improvement suggestion module is used to propose targeted improvement suggestions for the model design based on the shortcomings identified in the analysis. The model improvement module is used to improve large models based on improvement suggestions. The retest module is used to retest the improved large model with the same test set, in the same way as step S2, to compare the performance changes of the model before and after the improvement. The improvement effect evaluation module is used to compare the performance of the large model before and after the improvement to quantitatively evaluate the improvement effect, compare the improvement of the comprehensive score A, and use statistical methods to test the significance of the improvement effect.

Images