Large model fine-tuning training system

Abstract

This invention discloses a large-scale model fine-tuning training system, which belongs to the field of educational model training technology. In this system, the data acquisition subsystem is responsible for collecting datasets containing training and verification question-answer pairs related to the humanities and social sciences. These data are sourced from professional textbooks, academic papers, and internship cases. The model fine-tuning subsystem uses small-sample learning methods to train and fine-tune a selected general large-scale model, outputting the fine-tuned model. The fine-tuned model verification subsystem evaluates the performance of the fine-tuned model and, based on the evaluation results, determines whether further model optimization is needed to ensure the model's accuracy and practicality. This invention improves the accuracy and practicality of models in the field of education and teaching. By screening and grading data difficulty, it ensures that each step of data processing is aimed at improving the quality and efficiency of the final model, thereby making the model more adaptable to specific educational application scenarios and needs.

Description

Technical Field

[0001] The present invention belongs to the technical field of educational teaching model training, and specifically relates to a large model fine-tuning training system. Background Art

[0002] When general-purpose large language models are used in actual vertical business scenarios, their general capabilities can sometimes become an application barrier. This is because general-purpose models are typically trained on broad datasets rather than domain-specific data, which can lead to insufficient performance when handling complex problems in specific areas. Furthermore, general-purpose models lack a deep understanding of the knowledge, terminology, and processes specific to certain industries. For example, in the medical or financial fields, a precise understanding of specialized terminology and concepts is crucial for providing accurate answers, and general-purpose models may lack a deep understanding of these specialized fields.

[0003] Furthermore, general-purpose models may require extensive hardware resources to deploy and run, which can be an obstacle for certain application scenarios. Furthermore, general-purpose models may not be able to capture the latest domain information because their training data may not be updated in real time. These limitations suggest that while general-purpose large language models have achieved remarkable results on a variety of natural language processing tasks, they may need to be combined with domain-specific knowledge graphs in specialized question-answering systems to improve the accuracy and relevance of question answering.

[0004] The general model can be applied to large models related to education and teaching, such as the "AI Talent" education and teaching-specific large model. This large model aims to explore and try to solve the difficulties in the construction of new liberal arts in the past, such as the high technical threshold of the integration of arts and sciences, the disconnection between experiments and practicality, and the difficulty of integrating into the local ecology, through the deep integration of artificial intelligence technology and philosophy and social science higher education teaching. It will form the precipitation, accumulation, sharing, innovation and inheritance of knowledge data among eight disciplines such as economics, management, literature, law, science, engineering, education, and art, and help to cultivate high-quality top innovative talents in new finance.

[0005] Therefore, a new system for fine-tuning training of large models is needed to better adapt to specific professional requirements and actual situations such as education and teaching, and to solve the performance problems that traditional general models may encounter in fields such as education and teaching. Summary of the Invention

[0006] In response to the shortcomings of the existing technology, the present invention provides a large-model fine-tuning training system, which improves the accuracy and practicality of the model in the field of education and teaching, thereby effectively overcoming the performance deficiencies that traditional general models may encounter in the field of education and teaching.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a large model fine-tuning training system, including a data acquisition subsystem, a model fine-tuning subsystem and a post-fine-tuning model verification subsystem, wherein:

[0008] The data acquisition subsystem is used to obtain training data and verification data in the humanities and social sciences. The training data includes several training question-training answer pairs, and the verification data includes several verification question-verification answer pairs.

[0009] The model fine-tuning subsystem is used to determine the general large model to be fine-tuned, and train and fine-tune the determined general large model based on training data and small sample learning, and output the fine-tuned large model for use in the humanities and social sciences.

[0010] The fine-tuned model validation subsystem is used to evaluate the performance of the fine-tuned large model based on the validation data and determine whether the fine-tuned large model needs to be optimized based on the evaluation results:

[0011] If optimization is not required, the fine-tuned large model after verification is output;

[0012] If optimization is required, the initial parameters of the determined general large model are reset and transmitted to the model fine-tuning subsystem, which then re-fine-tunes the model based on the training data obtained in the humanities and social sciences.

[0013] Preferably, the obtaining of training data comprises the following steps:

[0014] Obtain data to be screened related to humanities and social sciences content. The data to be screened includes questions to be screened and the corresponding answers to the questions. Humanities and social sciences content includes professional textbooks, academic papers, and internship training cases related to economics, management, literature, law, science, engineering, education, and art.

[0015] Randomly group the data to be screened to obtain several data groups to be screened;

[0016] Performing intra-group screening on each data group to be screened to obtain a screened data group;

[0017] Determine the most similar filtered data group pair, perform secondary screening on the filtered data group pair, and obtain a secondary filtered data group;

[0018] The data group after secondary screening and the data group after screening without secondary screening are split to obtain training data.

[0019] Preferably, obtaining the screened data set comprises the following steps:

[0020] Randomly select any one of the questions to be screened in the data group to be screened, obtain the text keyword combination to be screened of the question to be screened, and save it to the text keyword set corresponding to the data group to be screened;

[0021] Obtain the text keywords for the next question to be screened in the data group to be screened one by one and record them as the text keyword combination to be compared;

[0022] Perform similarity comparisons on the text keyword combination to be compared with the text keyword combination to be filtered in the text keyword set one by one:

[0023] If the text keyword combination to be compared is similar to any text keyword combination to be screened in the text keyword set, then the text keyword combination to be compared is deleted and the question to be screened corresponding to the text keyword combination to be compared and the answer to the question to be screened are deleted;

[0024] If the text keyword combination to be compared is not similar to any text keyword combination to be screened in the text keyword set, the text keyword combination to be compared is saved in the text keyword set, and the question to be screened corresponding to the text keyword combination to be compared and the answer to the question to be screened are retained;

[0025] The data to be screened after the intra-group screening is recorded as a screened data group, and the text keyword set corresponding to the screened data group is saved.

[0026] Preferably, determining the most similar screened data pair, performing secondary screening on the screened data pair, and obtaining a secondary screened data set comprises the following steps:

[0027] Perform similarity judgment on the text keyword sets corresponding to any two filtered data groups:

[0028] If two text keyword sets are not similar, the corresponding two filtered data sets will not be screened again;

[0029] If two text keyword sets are similar, then the corresponding two filtered data groups are the most similar filtered data group pairs;

[0030] determining the number of to-be-screened questions in the two screened data groups in the most similar pair of screened data groups;

[0031] Using the larger filtered data set as a benchmark, the similarity between the unfiltered questions in the smaller filtered data set and the unfiltered questions in the larger filtered data set is compared:

[0032] If they are similar, the questions to be screened in the filtered data group with a smaller number will be deleted;

[0033] If they are not similar, the unscreened problems in the filtered data group with a small number of them are retained;

[0034] Output the two filtered data sets to obtain a secondary filtered data set.

[0035] Preferably, determining the general large model to be fine-tuned comprises the following steps:

[0036] The training data is grouped by difficulty level to obtain first difficulty training data, second difficulty training data, and third difficulty training data;

[0037] Input the training questions in the first difficulty training data, the second difficulty training data, and the third difficulty training data into each candidate universal large model one by one, and obtain the response evaluation data of each candidate universal large model, the response evaluation data including the first difficulty response evaluation data, the second difficulty response evaluation data, and the third difficulty response evaluation data;

[0038] Determine the general large model to be fine-tuned based on the response evaluation data.

[0039] Preferably, grouping the training data by difficulty level comprises the following steps:

[0040] Obtain the training question keyword combination corresponding to each training question in the training data;

[0041] Obtain the keyword combination sets of each difficulty level stored in the database, and determine whether the training question keyword combination hits one of the keyword combination sets of each difficulty level:

[0042] If there is a hit, the difficulty coefficient corresponding to the hit difficulty level keyword combination set stored in the database is obtained;

[0043] If there is no hit, the difficulty coefficient corresponding to the keyword combination of the training question is 0;

[0044] Get the accuracy rate of human responses to each training question;

[0045] Determine the difficulty distribution coefficient for each training question based on the difficulty coefficient and the correct answer rate:

[0046] ;

[0047] Where A is the difficulty distribution coefficient, Nx is the difficulty coefficient, DL is the correct rate of human response, and a is The weight factor of Nx stored in the database is The weight factor of DL stored in the database;

[0048] If the difficulty allocation coefficient is less than the first difficulty allocation threshold stored in the database, the training question-training answer pair corresponding to the difficulty allocation coefficient is the first difficulty training data;

[0049] If the difficulty allocation coefficient is not less than the first difficulty allocation threshold stored in the database, but less than the second difficulty allocation threshold stored in the database, then the training question-training answer pair corresponding to the difficulty allocation coefficient is second difficulty training data;

[0050] If the difficulty allocation coefficient is not less than the second difficulty allocation threshold stored in the database, the training question-training answer pair corresponding to the difficulty allocation coefficient is the third difficulty training data.

[0051] Preferably, determining the general large model to be fine-tuned based on the response evaluation data comprises the following steps:

[0052] Obtaining first difficulty response parameter data, second difficulty response parameter data, and third difficulty response parameter data stored in a database;

[0053] Based on the first difficulty answer evaluation data and the first difficulty answer parameter data, the first difficulty answer performance index corresponding to the candidate general large model is obtained. The first difficulty answer evaluation data includes the first difficulty answer accuracy rate, the first difficulty correct answer completeness rate, the first difficulty correct answer reaction time, and the first difficulty correct answer non-empty rate. The first difficulty answer parameter data includes the first difficulty parameter answer accuracy rate, the first difficulty parameter correct answer completeness rate, the first difficulty parameter correct answer reaction time, and the first difficulty parameter correct answer non-empty rate.

[0054] Based on the second difficulty answer evaluation data and the second difficulty answer parameter data, the second difficulty answer performance index corresponding to the candidate general large model is obtained, the second difficulty answer evaluation data includes the second difficulty answer accuracy rate, the second difficulty correct answer completeness rate, the second difficulty correct answer reaction time, and the second difficulty correct answer non-empty rate; the second difficulty answer parameter data includes the second difficulty parameter answer accuracy rate, the second difficulty parameter correct answer completeness rate, the second difficulty parameter correct answer reaction time, and the second difficulty parameter correct answer non-empty rate;

[0055] Based on the third difficulty response evaluation data and the third difficulty response parameter data, the third difficulty response performance index corresponding to the candidate general large model is obtained. The third difficulty response evaluation data includes the third difficulty response accuracy rate, the third difficulty correct answer completeness rate, the third difficulty correct answer reaction time, and the third difficulty correct answer non-empty rate. The third difficulty response parameter data includes the third difficulty parameterized response accuracy rate, the third difficulty parameterized correct answer completeness rate, the third difficulty parameterized correct answer reaction time, and the third difficulty parameterized correct answer non-empty rate.

[0056] The first difficulty response performance index, the second difficulty response performance index and the third difficulty response performance index are weighted and summed to obtain the overall performance effect value of the large model;

[0057] The candidate general large model corresponding to the smallest large model overall performance effect value is determined as the general large model to be fine-tuned.

[0058] Preferably, the training and fine-tuning of the determined general large model based on the training data and small sample learning, and outputting the fine-tuned large model, comprises the following steps:

[0059] Initialize the fine-tuning parameters of the general large model, including the learning rate, input training data batch size, and number of iterations;

[0060] Create task prompts and construct model inputs;

[0061] Load the determined general large model;

[0062] Calculate the loss based on the cross entropy loss function;

[0063] Based on the loss calculation results, the gradient descent algorithm adjusts the model weights through the Adam optimizer to reduce the loss value;

[0064] After completing the number of iterations, the fine-tuned large model is output.

[0065] Preferably, the performance evaluation of the fine-tuned large model based on the validation data comprises the following steps:

[0066] Input the verification questions in the verification data into the fine-tuned large model one by one to obtain the output answer set;

[0067] Based on the verification question-verification answer pair, the answer set is analyzed to obtain verification result data, which includes the verification result accuracy rate, the correct verification result completeness rate, and the correct verification result non-hole rate;

[0068] Obtaining hardware system operation data during the verification process, including the number of hardware system crashes during the verification process, energy consumption during the verification process, and memory usage during the verification process;

[0069] Acquire the verification result allowable deviation data, which includes the correct verification result completeness rate allowable deviation value and the correct verification result non-void rate allowable deviation value;

[0070] Obtaining verification requirement performance data stored in a database, the verification requirement performance data including verification result accuracy rate, correct verification result completeness rate, correct verification result non-void rate, required energy consumption during the verification process, and required memory usage during the verification process;

[0071] Determine whether the number of hardware system crashes during the verification process is 0:

[0072] If it is not 0, the evaluation result of the performance of the large model after fine-tuning is that it needs to be optimized;

[0073] If it is not 0, the verification index is obtained based on the verification result data, the hardware system operation data, the verification result allowable deviation data, and the verification requirement performance data;

[0074] If the validation index is greater than the validation threshold stored in the database, the performance evaluation result of the fine-tuned large model is that it needs to be optimized;

[0075] If the validation index is not greater than the validation threshold stored in the database, the evaluation result of the performance of the large model after fine-tuning is that no optimization is required.

[0076] Preferably, obtaining the verification result allowable deviation data includes the following steps:

[0077] Obtaining a first difficulty response performance index, a second difficulty response performance index, and a third difficulty response performance index and recording them as comparison data;

[0078] Acquire various matching data stored in a database, the matching data including a first difficulty response performance index matching value, a second difficulty response performance index matching value, and a third difficulty response performance index matching value;

[0079] Compare the comparison data with the matching data one by one to determine the matching data that is most similar to the comparison data;

[0080] The most similar matching data is obtained, and the verification result tolerance data stored in the database is obtained.

[0081] The present invention has the following beneficial effects:

[0082] By using high-quality, specialized training and validation data drawn from professional textbooks, academic papers, and real-world cases, this method refines and optimizes the model's learning capabilities, making it better suited to the professional requirements and practical situations of education and teaching. This approach improves the model's accuracy and practicality in education and teaching, effectively overcoming the performance deficiencies that can arise from traditional general-purpose models in educational applications.

[0083] By screening and grading data difficulty, this method ensures that every step of data processing is designed to improve the quality and efficiency of the final model, thereby making the model more adaptable to educational application scenarios and needs. This strategy is particularly suitable for complex model scenarios where the most useful information related to education and teaching must be extracted from large amounts of data for efficient training. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 Schematic diagram of the subsystem structure of the system of the present invention;

[0085] Figure 2 Flowchart for determining a general macro model to be fine-tuned for the present invention. DETAILED DESCRIPTION

[0086] The embodiments of the present invention are further described below with reference to the accompanying drawings:

[0087] like Figure 1 As shown, the large model fine-tuning training system includes a data acquisition subsystem, a model fine-tuning subsystem and a post-fine-tuning model verification subsystem, wherein: the data acquisition subsystem is used to obtain training data and verification data in the humanities and social sciences fields, the training data includes several training question-training answer pairs, and the verification data includes several verification question-verification answer pairs.

[0088] Obtain data to be screened related to content in the humanities and social sciences. This data contains question and answer pairs within professional fields (economics, management, literature, law, science, engineering, education, art, and other disciplines), providing the model with application cases and professional knowledge in real-world scenarios. The data to be screened include questions to be screened and the answers to the questions to be screened. It should be noted that the data to be screened can be subdivided into economics-related data to be screened, management-related data to be screened, and literature-related data to be screened, etc. These data to be screened can be obtained from professional textbooks, academic papers, and internship training cases in various humanities and social sciences fields; the data to be screened are randomly grouped to obtain several data groups to be screened. It should be noted that random grouping can group economics-related data to be screened or other subject-related data to be screened into similar subjects, or all subject-related data to be screened can be grouped together so that the grouped data to be screened can contain content from multiple subjects; each data group to be screened is screened within the group to obtain a screened data group; each data group is independently screened based on standards such as data quality and relevance, which helps to extract the most representative and practical data from each group.

[0089] Identify the most similar pairs of filtered data sets and perform secondary screening on these pairs to obtain secondary filtered data sets. Compare the different filtered data sets to identify the most similar ones for further refined screening. This step further refines the data, ensuring the resulting training data sets are highly relevant to the specific field and valuable for learning. Split the secondary filtered data sets into the non-secondary filtered data sets to obtain training data. Use the secondary filtered and non-secondary filtered data sets separately for model training. This partitioning maximizes data utilization during model training while maintaining the broad coverage and depth of the dataset.

[0090] Through a multi-stage screening process, the system is able to select the most relevant and informative data from a large amount of candidate data related to the humanities and social sciences. This ensures the quality of the training data, thereby improving the efficiency of the model in question-answering training in the humanities and social sciences and the final model performance. Through random grouping and within-group screening, data diversity is ensured, helping the model learn a wider range of problem-solving strategies and enhancing its generalization ability in applications in humanities and social sciences education.

[0091] Careful data selection can reduce noise and irrelevant information in the data, which helps prevent the model from overfitting to the training data during training (i.e., overfitting) and failing to perform well on new, unseen data. Precisely controlling the selection and use of data allows for more efficient use of computing resources, which is particularly important when the data volume is large and the computational cost is high.

[0092] Randomly select any question to be screened in the data group to be screened, obtain the text keyword combination to be screened for the question to be screened and save it to the text keyword set corresponding to the data group to be screened; randomly select a question from the data group to be screened as the starting point, and extract keywords. Such randomness helps to avoid initial bias and ensure the representativeness of the keyword set.

[0093] The system then retrieves the keywords for the next question in the dataset and records them as the keyword combination to be compared. By comparing the keywords for each new question with the existing keyword set, the system can effectively identify and eliminate duplicate or highly similar questions. This step ensures that each question is unique, increasing the information entropy of the training data.

[0094] The similarity of the text keyword combination to be compared is compared with the text keyword combination to be screened in the text keyword set one by one: if the text keyword combination to be compared is similar to any text keyword combination to be screened in the text keyword set, the text keyword combination to be compared is deleted and the corresponding question to be screened and the answer to the question to be screened are deleted; if the text keyword combination to be compared is not similar to any text keyword combination to be screened in the text keyword set, the text keyword combination to be compared is saved in the text keyword set, and the corresponding question to be screened and the answer to the question to be screened are retained; whenever a new and unique keyword combination is discovered, it is added to the set. This continuous updating process helps to maintain the freshness and relevance of the dataset.

[0095] After the group is filtered, the data to be filtered is recorded as the filtered data group, and the text keyword set corresponding to the filtered data group is saved. If the keyword combination of a question is considered duplicate, the related questions and answers will be deleted to avoid wasting resources on duplicate or similar questions. If the keyword combination is unique, the question and answer will be retained, providing more data angles for model training.

[0096] By removing questions and answers with similar keyword combinations, the system reduces data redundancy, ensuring that each retained data point contributes independent value to model training. This not only improves the quality of the training data but also enhances the model's ability to handle new questions. Retaining questions with dissimilar keyword combinations increases the diversity of the dataset. This diversity is crucial for building a robust model, ensuring it performs more reliably across different types of questions.

[0097] The similarity of the text keyword sets corresponding to any two filtered data groups is judged: if the two text keyword sets are not similar, the corresponding two filtered data groups are not subjected to secondary screening; if the two text keyword sets are similar, the corresponding two filtered data groups are the most similar filtered data group pair; the number of questions to be screened in the two filtered data groups in the most similar filtered data group pair is determined; the filtered data group with a larger number is used as a benchmark, and the questions to be screened in the other filtered data group with a smaller number are compared for similarity (through the combination of text keywords to be screened corresponding to the questions to be screened): if they are similar, the questions to be screened in the filtered data group with a smaller number are deleted; if they are not similar, the questions to be screened in the filtered data group with a smaller number are retained; the two filtered data groups are output to obtain a secondary filtered data group.

[0098] By performing a secondary screening of the most similar data groups, we can further streamline the dataset, removing duplicate or overly similar data, and ensuring that the model training set contains the most representative and useful information. By reducing the training of duplicate data, we can save computing resources and time, making the training process more efficient.

[0099] The model fine-tuning subsystem identifies a general large model to be fine-tuned. It then trains and fine-tunes this large model based on training data and small-sample learning, outputting the fine-tuned large model. Small-sample learning methods are used to perform targeted training on the pre-trained general large model. This step uses specialized training data to adjust the model's parameters, enabling the model to better understand and address specific problems in a specialized domain.

[0100] like Figure 2As shown, the training data are grouped by difficulty level to obtain first difficulty training data, second difficulty training data and third difficulty training data; the training questions in the first difficulty training data, the second difficulty training data and the third difficulty training data are input into each candidate universal large model one by one, and the response evaluation data of each candidate universal large model is obtained, and the response evaluation data includes first difficulty response evaluation data, second difficulty response evaluation data and third difficulty response evaluation data; the universal large model to be fine-tuned is determined based on the response evaluation data.

[0101] By grading the difficulty of training data, we can see how different candidate general-purpose large-scale models perform on humanities and social science question-answering tasks before fine-tuning, thereby identifying suitable general-purpose large-scale models for fine-tuning. Furthermore, grading the difficulty of training data allows for more targeted model training. Models can gradually learn from data of varying difficulty, moving from simple to complex, and gradually adapting to more difficult problems. This progressive training helps improve the model's learning efficiency and ultimate performance.

[0102] Grouping the training data by difficulty level includes the following steps: obtaining the training question keyword combination corresponding to each training question in the training data; obtaining the keyword combination sets of each difficulty level stored in the database (relevant professionals, such as on-the-job professors, rate the difficulty of different keyword combinations, and construct keyword combinations of the same difficulty level into keyword combination sets of each difficulty level); determining whether the training question keyword combination hits one of the difficulty level keyword combination sets (keyword similarity matching): if so, obtaining the difficulty coefficient stored in the database corresponding to the hit difficulty level keyword combination set; if not, the difficulty coefficient corresponding to the training question keyword combination is 0; obtaining the correct human response rate for each training question (the correct response rate for each question can be collected through a large number of written or online tests). Combined with the actual response accuracy rate, the difficulty level of the question can be further adjusted or used to evaluate the performance of the model in practical applications.

[0103] Determine the difficulty distribution coefficient for each training question based on the difficulty coefficient and the correct answer rate:

[0104] ;

[0105] Where A is the difficulty distribution coefficient, Nx is the difficulty coefficient, DL is the correct rate of human response, and a is The weight factor of Nx stored in the database is The weight factor of DL stored in the database.

[0106] If the difficulty distribution coefficient is less than the first difficulty distribution threshold stored in the database, the training question-training answer pair corresponding to the difficulty distribution coefficient is the first difficulty training data; if the difficulty distribution coefficient is not less than the first difficulty distribution threshold stored in the database, but less than the second difficulty distribution threshold stored in the database, the training question-training answer pair corresponding to the difficulty distribution coefficient is the second difficulty training data; if the difficulty distribution coefficient is not less than the second difficulty distribution threshold stored in the database, the training question-training answer pair corresponding to the difficulty distribution coefficient is the third difficulty training data.

[0107] Before performing any form of fine-tuning, it's crucial to understand how the model responds to and solves problems of varying difficulty in its untuned state. This provides a clear baseline to help assess the effectiveness and necessity of fine-tuning. By comparing the performance of different candidate models on training data at varying difficulty levels, those that perform best on a specific problem set can be selected for further fine-tuning. This not only improves the efficiency of the fine-tuned model but also ensures that resources are invested in the candidates most likely to succeed.

[0108] Obtain the first difficulty response parameter data, the second difficulty response parameter data, and the third difficulty response parameter data stored in the database. Finally, based on the response evaluation data, select an appropriate general large model to be fine-tuned to ensure that the selected model can perform well when dealing with practical problems.

[0109] Based on the first difficulty answer evaluation data and the first difficulty answer parameter data, the first difficulty answer performance index corresponding to the candidate general large model is obtained. The first difficulty answer evaluation data includes the first difficulty answer accuracy rate (the ratio of the number of correctly answered questions to the total number of questions), the first difficulty correct answer completeness rate (the ratio of the number of keywords in the correct answers to the required number of keywords), the first difficulty correct answer reaction time (the time from the completion of question input to the start of answer output) and the first difficulty correct answer non-empty rate (the ratio of the number of correct keyword words to the total number of output words). The first difficulty answer parameter data includes the first difficulty parameter answer accuracy rate, the first difficulty parameter correct answer completeness rate, the first difficulty parameter correct answer reaction time and the first difficulty parameter correct answer non-empty rate.

[0110] ;

[0111] Where i is the number of the candidate general large model, Fzs i is the correct answer rate of the first difficulty of the i-th candidate general large model, Fws i is the completeness rate of the correct answer of the first difficulty of the i-th candidate general large model, Fts i Fks is the reaction time of the first difficulty correct answer of the i-th candidate general large model,i is the non-void rate of the first difficulty correct answer of the i-th candidate general large model, Fzc is the correct rate of the first difficulty parameter answer, Fwc is the complete rate of the first difficulty parameter correct answer, Ftc is the reaction time of the first difficulty parameter correct answer, Fkc is the non-void rate of the first difficulty parameter correct answer, Fr i is the first difficulty response performance index of the i-th candidate general large model.

[0112] Based on the second difficulty answer evaluation data and the second difficulty answer parameter data, the second difficulty answer performance index corresponding to the candidate general large model is obtained. The second difficulty answer evaluation data includes the second difficulty answer accuracy rate, the second difficulty correct answer completeness rate, the second difficulty correct answer reaction time and the second difficulty correct answer non-empty rate. The second difficulty answer parameter data includes the second difficulty parameter answer accuracy rate, the second difficulty parameter correct answer completeness rate, the second difficulty parameter correct answer reaction time and the second difficulty parameter correct answer non-empty rate.

[0113] ;

[0114] Where Szs i is the correct answer rate of the second difficulty of the i-th candidate general large model, Sws i is the completeness rate of the correct answer of the second difficulty of the i-th candidate general large model, Sts i Sks is the reaction time of the second difficulty correct answer of the i-th candidate general large model, i is the non-empty rate of the second difficulty correct answer of the i-th candidate general large model, Szc is the correct rate of the second difficulty parameter answer, Swc is the complete rate of the second difficulty parameter correct answer, Stc is the reaction time of the second difficulty parameter correct answer, Skc is the non-empty rate of the second difficulty parameter correct answer, Sr i is the second difficulty response performance index of the i-th candidate general large model.

[0115] Based on the third difficulty answer evaluation data and the third difficulty answer parameter data, the third difficulty answer performance index corresponding to the candidate general large model is obtained. The third difficulty answer evaluation data includes the third difficulty answer accuracy rate, the third difficulty correct answer completeness rate, the third difficulty correct answer reaction time and the third difficulty correct answer non-empty rate. The third difficulty answer parameter data includes the third difficulty parameter answer accuracy rate, the third difficulty parameter correct answer completeness rate, the third difficulty parameter correct answer reaction time and the third difficulty parameter correct answer non-empty rate.

[0116] ;

[0117] Where Tzs iis the correct answer rate of the third difficulty of the i-th candidate general large model, Tws i is the completeness rate of the correct answer of the third difficulty level of the i-th candidate general large model, Tts i Tks is the reaction time of the correct answer of the third difficulty level of the i-th candidate general large model, i is the non-empty rate of the correct answer of the third difficulty parameter of the i-th candidate general large model, Tzc is the correct rate of the third difficulty parameter answer, Twc is the complete rate of the correct answer of the third difficulty parameter, Ttc is the reaction time of the correct answer of the third difficulty parameter, Tkc is the non-empty rate of the correct answer of the third difficulty parameter, Tr i is the second difficulty response performance index of the i-th candidate general large model.

[0118] By comprehensively considering accuracy, answer completeness, response speed, and answer quality (non-empty rate), this formula comprehensively assesses a model's overall ability to handle real-world problems, rather than simply accuracy. The parameters in the formula serve as benchmarks, ensuring that models are evaluated relative to a set of pre-set standards. This facilitates fair comparison of performance across models. By calculating each model's response performance index, the best performing models at the first difficulty level can be intuitively compared and selected for further fine-tuning and deployment, thereby improving the quality of the final product and user satisfaction.

[0119] The natural logarithm is used in the formula to smooth the impact of the indicators and prevent a single extreme value from overly influencing the overall score. Logarithmic transformation can help reduce distortion caused by extreme values, making the score more stable and reasonable. Each comparison term in the formula effectively normalizes the model's performance against a preset benchmark (reference data). This comparison helps maintain consistency and comparability across different tests.

[0120] A high accuracy rate is a fundamental indicator of good model performance, but it needs to be combined with other parameters to assess the completeness and quality of the answer. Completeness rate complements the limitations of accuracy rate, ensuring that the model not only provides correct answers but also provides comprehensive information. Fast responses must balance accuracy and completeness, ensuring that speed does not sacrifice answer quality. Non-voidness rate emphasizes improving the density and quality of information while maintaining accuracy and completeness, avoiding lengthy and ineffective responses.

[0121] Together, these parameters form a multi-dimensional model evaluation framework that considers not only the basic accuracy of the model's answers but also the quality, efficiency, and information density of the answers. This comprehensive evaluation provides a more comprehensive understanding of the model's performance in real-world applications, ensuring that the selected model not only responds quickly but also provides accurate, informative, and practical answers. This evaluation approach helps identify and optimize potential model deficiencies, enabling it to better adapt to specific application needs.

[0122] The first difficulty response performance index, the second difficulty response performance index and the third difficulty response performance index are weighted and summed to obtain the overall performance effect value of the large model; the candidate general large model corresponding to the smallest large model overall performance effect value is determined as the general large model to be fine-tuned.

[0123] ;

[0124] Where Zx i is the overall performance value of the large model of the i-th candidate general large model, and c1 is the Fr stored in the database i The weight factor of Sr is stored in the database. i The weight factor, c3 is the Tr stored in the database i The weight factors of c1, c2 and c3 are summed to 1.

[0125] Initialize the fine-tuning parameters of the determined general large model, including the learning rate, input training data batch size, and number of iterations; create task prompts and build model inputs; load the determined general large model; calculate the loss based on the cross-entropy loss function; based on the loss calculation results, the gradient descent algorithm uses the Adam optimizer to adjust the model weights to reduce the loss value; after completing the number of iterations, output the fine-tuned large model.

[0126] In one specific implementation, the goal is to fine-tune a pre-trained large model (e.g., BERT) using six question-answer pairs to improve its performance on the question-answering task. The specific fine-tuning steps are as follows:

[0127] There are 6 sets of question-answer pairs that constitute the fine-tuning dataset:

[0128] Question: "What is the central question studied in economics?" Answer: "The allocation of resources."

[0129] Question: "What does the law of diminishing marginal utility refer to?" Answer: "As the amount of goods consumed increases, the satisfaction derived from each additional unit decreases."

[0130] Question: "What is the definition of art?" Answer: "Art is the creation of images with aesthetic value through the reflection and processing of real life to meet people's spiritual needs."

[0131] Question: "What is feedforward control in management?" Answer: "Feedforward control, also known as pre-emptive control or advance control, refers to an organization's prediction and estimation of potential deviations in work before work activities officially begin, and the implementation of preventive measures to eliminate potential deviations before they occur."

[0132] Question: "What is quantum mechanics?" Answer: "Quantum mechanics is a fundamental theory in physics that describes natural phenomena at the smallest energy levels of atoms and subatomic particles."

[0133] Question: "What is the definition of education?" Answer: "Education refers to a planned, purposeful, and organized social activity that promotes the all-round development of students by imparting knowledge, cultivating abilities, and shaping moral character."

[0134] The training parameters include: batch size: 2; number of iterations: 3; learning rate: 5e-5 (a low learning rate commonly used for fine-tuning); optimizer: AdamW; loss function: cross-entropy loss function, which is used to evaluate the difference between the model output and the correct answer.

[0135] Fine-tuning steps:

[0136] Step 1: Prepare the model and data.

[0137] Loading a pre-trained model: Start with a large pre-trained model (such as BERT) and load the pre-trained weights.

[0138] Data preprocessing: Encode the six question-answer pairs to fit the model's input format. Questions and answers will be converted into an input form acceptable to the model (usually vectorized or tokenized).

[0139] Step 2: First iteration.

[0140] Batch 1 input: The model first receives the first two sets of question-answer pairs.

[0141] Forward propagation: Based on the input question and answer, the model predicts what it thinks is the correct answer and outputs a probability distribution.

[0142] Calculating Loss: The loss function (cross entropy loss) is used to compare the model's predicted output with the actual answer. The loss value here reflects the difference between the model and the correct answer, with larger values indicating less accurate model predictions.

[0143] Backpropagation and weight updates: Based on the loss calculation, the gradient descent algorithm uses the Adam optimizer to adjust the model's weights to reduce the loss. This is how the model "learns," gradually adjusting itself to get closer to the correct answer next time.

[0144] Batch 2 input: The model then processes the next two sets of question-answer pairs.

[0145] The steps repeat: the model predicts the answer again, calculates the loss, performs backpropagation, and updates the weights.

[0146] Batch 3 input: The last batch of data. Process the last two question-answer pairs, calculate the loss, and update the model weights.

[0147] In the first iteration, the model processed all 6 sets of question-answer pairs (divided into 3 batches of 2 pairs each), and the weight updates in each batch helped the model learn how to better answer these questions.

[0148] Step 3: Second iteration. The second iteration further optimizes the model's performance and continues to reduce the loss value by repeatedly processing the same question-answer pairs.

[0149] Step 4: The third iteration.

[0150] After three iterations of training, the fine-tuned model has learned enough information from the six question-answer pairs to accurately answer these specific questions. At this point, the model weights have been optimized, and the fine-tuned model performs better on question-answering tasks than the unfine-tuned pre-trained model.

[0151] The fine-tuned model validation subsystem evaluates the performance of the fine-tuned large model based on validation data and, based on the evaluation results, determines whether the fine-tuned large model requires optimization. If not, the verified fine-tuned large model is output. If optimization is required, the initial parameters of the general large model are reset and fed to the model fine-tuning subsystem, which then fine-tunes it based on acquired training data from the humanities and social sciences. The fine-tuned model is then evaluated for performance using specialized validation data. Based on the evaluation results, the system determines whether further model optimization is necessary, such as adjusting initial parameters and re-fine-tuning, to achieve higher accuracy and efficiency.

[0152] Input the verification questions in the verification data one by one into the fine-tuned large model to obtain the output answer set; analyze the answer set based on the verification question-verification answer pair to obtain the verification result data, which includes the verification result accuracy, the correct verification result completeness rate and the correct verification result non-void rate; obtain the hardware system operation data during the verification process, which includes the number of hardware system crashes during the verification process, the energy consumption during the verification process and the memory usage during the verification process; obtain the verification result allowable deviation data, which includes the correct verification result completeness rate allowable deviation value and the correct verification result non-void rate allowable deviation value; obtain the verification requirement performance data stored in the database, which includes the verification result requirement accuracy rate, the correct verification result requirement completeness rate, the correct verification result requirement non-void rate, the required energy consumption during the verification process and the required memory usage during the verification process.

[0153] By combining the model's answer accuracy, completeness, and non-void rate with hardware performance (energy consumption and memory usage), this evaluation framework comprehensively considers the model's performance in real-world environments, ensuring its practicality and reliability. Energy consumption is related to the actual deployment cost and sustainability of the model. This, along with performance metrics such as accuracy and completeness, ensures that the model maintains reasonable resource usage while meeting performance requirements. Memory usage is correlated with energy consumption; together, they reflect the model's efficiency and adaptability to physical resources.

[0154] Together, these parameters form a comprehensive evaluation system that considers both the model's theoretical problem-solving ability (measured by accuracy, completeness, and non-voidness ratio) and its resource efficiency and stability in practical deployments (measured by energy consumption and memory usage). This multi-dimensional evaluation ensures that the model is efficient, stable, and cost-effective in practical applications, while providing a method for balancing model performance and resource consumption.

[0155] Determine whether the number of hardware system crashes during the verification process is 0: If it is not 0, the evaluation result of the performance of the large model after fine-tuning is that optimization is required; if it is not 0, obtain the verification index based on the verification result data, hardware system operation data, verification result allowable deviation data and verification requirement performance data.

[0156] ;

[0157] Where Y is the verification index, f1 is the first transfer function, f2 is the second transfer function, zq is the accuracy of the verification result, zw is the completeness rate of the correct verification result, zf is the non-void rate of the correct verification result, zn is the energy consumption during the verification process, and zN is the memory usage during the verification process. In order to correctly verify the completeness of the result, the deviation value is allowed. is the allowed deviation value of the non-void rate of the correct verification result, cq is the required accuracy rate of the verification result, sw is the required completeness rate of the correct verification result, sf is the required non-void rate of the correct verification result, cn is the required energy consumption during the verification process, and cN is the required memory usage during the verification process.

[0158] The f1 part evaluates the deviation of the model's response accuracy, completeness, and non-voidity rate from the requirements. This comparison measures the relative difference in performance by dividing the deviation between the actual value and the target value by the allowed deviation, which helps to quantitatively analyze the quality of the model's answers. Absolute values and deviation values are used in f1 to quantify the difference between actual performance and expected goals. Even small deviations are identified and taken into account. f2 focuses on the comparison of energy consumption and memory usage with preset standards, which directly reflects the resource efficiency of the model. The final verification index is determined by combining the outputs of the two transit functions. If the verification index exceeds the predetermined threshold, it means that the model's performance does not meet the requirements and needs further optimization.

[0159] If the verification index is greater than the verification threshold stored in the database, the evaluation result of the performance of the large model after fine-tuning is that optimization is required; if the verification index is not greater than the verification threshold stored in the database, the evaluation result of the performance of the large model after fine-tuning is that optimization is not required.

[0160] Obtain a first difficulty response performance index, a second difficulty response performance index and a third difficulty response performance index and record them as comparison data; obtain each matching data stored in a database, the matching data including a first difficulty response performance index matching value, a second difficulty response performance index matching value and a third difficulty response performance index matching value; compare the comparison data with the matching data one by one to determine the matching data that is most similar to the comparison data (the similarity value can be calculated using Euclidean distance); obtain the verification result allowable deviation data stored in the database corresponding to the most similar matching data.

[0161] In real-world applications, model outputs may fluctuate due to the diversity of input data and environmental changes. Setting a tolerance for bias provides flexibility, ensuring that the model remains consistent in the face of unforeseen data or environmental changes. Unrestricted bias can lead to over-optimization of specific data points or over-tuning the model to meet very strict standards, thereby neglecting the model's generalization ability. Allowing bias ensures that model optimization remains within a reasonable range, promoting good performance on unseen data.

[0162] The response performance indices for the first, second, and third difficulty levels provide actual performance data for the model at different difficulty levels. Adjusting the allowed deviation based on this data ensures that the model has appropriate expected standards for questions of different difficulty levels. By analyzing the performance indices at different difficulty levels, the expected performance of the model at each level can be calibrated more finely. For example, if a model performs well on high-difficulty questions, smaller performance fluctuations may be allowed on low-difficulty questions. The response performance index covers multiple aspects such as accuracy, completeness, and non-voidness rate. By comprehensively evaluating the performance of the model through these indices, we can have a more comprehensive understanding of the strengths and weaknesses of the model and set a reasonable deviation range accordingly, aiming to optimize the overall performance of the model rather than a single indicator.

[0163] This implementation also includes:

[0164] An electronic device includes: a processor; and a memory storing computer program instructions that, when executed by the processor, cause the processor to execute the large model fine-tuning training system described above. A computer-readable storage medium stores a program that, when executed by the processor, implements the large model fine-tuning training system described above.

Claims

1. Large model fine-tuning training system, characterized by: It includes data acquisition subsystem, model fine-tuning subsystem and post-fine-tuning model verification subsystem, among which: The data acquisition subsystem is used to obtain training data and verification data in the humanities and social sciences. The training data includes several training question-training answer pairs, and the verification data includes several verification question-verification answer pairs. The model fine-tuning subsystem is used to determine the general large model to be fine-tuned, and train and fine-tune the determined general large model based on training data and small sample learning, and output the fine-tuned large model for use in the humanities and social sciences. The fine-tuned model validation subsystem is used to evaluate the performance of the fine-tuned large model based on the validation data and determine whether the fine-tuned large model needs to be optimized based on the evaluation results: If optimization is not required, the fine-tuned large model after verification is output; If optimization is required, the initial parameters of the determined general large model are reset and transmitted to the model fine-tuning subsystem, which then fine-tunes the model based on the training data obtained in the humanities and social sciences. The step of obtaining training data comprises the following steps: Obtain data to be screened related to humanities and social sciences content. The data to be screened includes questions to be screened and the corresponding answers to the questions. Humanities and social sciences content includes professional textbooks, academic papers, and internship training cases related to economics, management, literature, law, science, engineering, education, and art. Randomly group the data to be screened to obtain several data groups to be screened; Performing intra-group screening on each data group to be screened to obtain a screened data group includes the following steps: Randomly select any one of the questions to be screened in the data group to be screened, obtain the text keyword combination to be screened of the question to be screened, and save it to the text keyword set corresponding to the data group to be screened; Obtain the text keywords for the next question to be screened in the data group to be screened one by one and record them as the text keyword combination to be compared; Perform similarity comparisons on the text keyword combination to be compared with the text keyword combination to be filtered in the text keyword set one by one: If the text keyword combination to be compared is similar to any text keyword combination to be screened in the text keyword set, then the text keyword combination to be compared is deleted and the question to be screened corresponding to the text keyword combination to be compared and the answer to the question to be screened are deleted; If the text keyword combination to be compared is not similar to any text keyword combination to be screened in the text keyword set, the text keyword combination to be compared is saved in the text keyword set, and the question to be screened corresponding to the text keyword combination to be compared and the answer to the question to be screened are retained; The data to be screened after the intra-group screening is recorded as a screened data group, and the text keyword set corresponding to the screened data group is saved; Determining the most similar filtered data pair, performing secondary screening on the filtered data pair, and obtaining a secondary filtered data group, comprises the following steps: Perform similarity judgment on the text keyword sets corresponding to any two filtered data groups: If two text keyword sets are not similar, the corresponding two filtered data sets will not be screened again; If two text keyword sets are similar, then the corresponding two filtered data groups are the most similar filtered data group pairs; determining the number of to-be-screened questions in the two screened data groups in the most similar pair of screened data groups; Using the larger filtered data set as a benchmark, the similarity between the unfiltered questions in the smaller filtered data set and the unfiltered questions in the larger filtered data set is compared: If they are similar, the questions to be screened in the filtered data group with a smaller number will be deleted; If they are not similar, the unscreened problems in the filtered data group with a small number of them are retained; Outputting the two filtered data groups to obtain a secondary filtered data group; The data group after secondary screening and the data group after screening without secondary screening are split to obtain training data.

2. The large model fine-tuning training system according to claim 1, characterized in that: Determining the general large model to be fine-tuned includes the following steps: The training data is grouped by difficulty level to obtain first difficulty training data, second difficulty training data, and third difficulty training data; Input the training questions in the first difficulty training data, the second difficulty training data, and the third difficulty training data into each candidate universal large model one by one, and obtain the response evaluation data of each candidate universal large model, the response evaluation data including the first difficulty response evaluation data, the second difficulty response evaluation data, and the third difficulty response evaluation data; Determine the general large model to be fine-tuned based on the response evaluation data.

3. The large model fine-tuning training system according to claim 2, characterized in that: The step of grouping the training data by difficulty level comprises the following steps: Obtain the training question keyword combination corresponding to each training question in the training data; Obtain the keyword combination sets of each difficulty level stored in the database, and determine whether the training question keyword combination hits one of the keyword combination sets of each difficulty level: If there is a hit, the difficulty coefficient corresponding to the hit difficulty level keyword combination set stored in the database is obtained; If there is no hit, the difficulty coefficient corresponding to the keyword combination of the training question is 0; Get the accuracy rate of human responses to each training question; Determine the difficulty distribution coefficient for each training question based on the difficulty coefficient and the correct answer rate: ； Where A is the difficulty distribution coefficient, Nx is the difficulty coefficient, DL is the correct rate of human response, and a is The weight factor of Nx stored in the database is The weight factor of DL stored in the database; If the difficulty allocation coefficient is less than the first difficulty allocation threshold stored in the database, the training question-training answer pair corresponding to the difficulty allocation coefficient is the first difficulty training data; If the difficulty allocation coefficient is not less than the first difficulty allocation threshold stored in the database, but less than the second difficulty allocation threshold stored in the database, then the training question-training answer pair corresponding to the difficulty allocation coefficient is second difficulty training data; If the difficulty allocation coefficient is not less than the second difficulty allocation threshold stored in the database, the training question-training answer pair corresponding to the difficulty allocation coefficient is the third difficulty training data.

4. The large model fine-tuning training system according to claim 2, characterized in that: The method of determining the general large model to be fine-tuned based on the response evaluation data includes the following steps: Obtaining first difficulty response parameter data, second difficulty response parameter data, and third difficulty response parameter data stored in a database; Based on the first difficulty answer evaluation data and the first difficulty answer parameter data, the first difficulty answer performance index corresponding to the candidate general large model is obtained. The first difficulty answer evaluation data includes the first difficulty answer accuracy rate, the first difficulty correct answer completeness rate, the first difficulty correct answer reaction time, and the first difficulty correct answer non-empty rate. The first difficulty answer parameter data includes the first difficulty parameter answer accuracy rate, the first difficulty parameter correct answer completeness rate, the first difficulty parameter correct answer reaction time, and the first difficulty parameter correct answer non-empty rate. The correct answer completeness rate for the first difficulty level is the ratio of the number of keywords in the correct answer to the number of required keywords. The correct answer response time for the first difficulty level is the time from the completion of question input to the start of answer output. The correct answer non-empty rate for the first difficulty level is the ratio of the number of correct keyword words to the total number of words output. Based on the second difficulty answer evaluation data and the second difficulty answer parameter data, the second difficulty answer performance index corresponding to the candidate general large model is obtained, the second difficulty answer evaluation data includes the second difficulty answer accuracy rate, the second difficulty correct answer completeness rate, the second difficulty correct answer reaction time, and the second difficulty correct answer non-empty rate; the second difficulty answer parameter data includes the second difficulty parameter answer accuracy rate, the second difficulty parameter correct answer completeness rate, the second difficulty parameter correct answer reaction time, and the second difficulty parameter correct answer non-empty rate; Based on the third difficulty response evaluation data and the third difficulty response parameter data, the third difficulty response performance index corresponding to the candidate general large model is obtained. The third difficulty response evaluation data includes the third difficulty response accuracy rate, the third difficulty correct answer completeness rate, the third difficulty correct answer reaction time, and the third difficulty correct answer non-empty rate. The third difficulty response parameter data includes the third difficulty parameterized response accuracy rate, the third difficulty parameterized correct answer completeness rate, the third difficulty parameterized correct answer reaction time, and the third difficulty parameterized correct answer non-empty rate. The first difficulty response performance index, the second difficulty response performance index and the third difficulty response performance index are weighted and summed to obtain the overall performance effect value of the large model; The candidate general large model corresponding to the smallest large model overall performance effect value is determined as the general large model to be fine-tuned.

5. The large model fine-tuning training system according to claim 4, characterized in that: The method of training and fine-tuning the determined general large model based on the training data and small sample learning, and outputting the fine-tuned large model, includes the following steps: Initialize the fine-tuning parameters of the general large model, including the learning rate, input training data batch size, and number of iterations; Create task prompts and construct model inputs; Load the determined general large model; Calculate the loss based on the cross entropy loss function; Based on the loss calculation results, the gradient descent algorithm adjusts the model weights through the Adam optimizer to reduce the loss value; After completing the number of iterations, the fine-tuned large model is output.

6. The large model fine-tuning training system according to claim 4, characterized in that: The performance evaluation of the fine-tuned large model based on the validation data includes the following steps: Input the verification questions in the verification data into the fine-tuned large model one by one to obtain the output answer set; Based on the verification question-verification answer pair, the answer set is analyzed to obtain verification result data, which includes the verification result accuracy rate, the correct verification result completeness rate, and the correct verification result non-hole rate; Obtaining hardware system operation data during the verification process, including the number of hardware system crashes during the verification process, energy consumption during the verification process, and memory usage during the verification process; Acquire the verification result allowable deviation data, which includes the correct verification result completeness rate allowable deviation value and the correct verification result non-void rate allowable deviation value; Obtaining verification requirement performance data stored in a database, the verification requirement performance data including verification result accuracy rate, correct verification result completeness rate, correct verification result non-void rate, required energy consumption during the verification process, and required memory usage during the verification process; Determine whether the number of hardware system crashes during the verification process is 0: If it is not 0, the evaluation result of the performance of the large model after fine-tuning is that it needs to be optimized; If it is not 0, the verification index is obtained based on the verification result data, the hardware system operation data, the verification result allowable deviation data, and the verification requirement performance data; If the validation index is greater than the validation threshold stored in the database, the performance evaluation result of the fine-tuned large model is that it needs to be optimized; If the validation index is not greater than the validation threshold stored in the database, the evaluation result of the performance of the large model after fine-tuning is that no optimization is required.

7. The large model fine-tuning training system according to claim 6, characterized in that: The method of obtaining the verification result allowable deviation data comprises the following steps: Obtaining a first difficulty response performance index, a second difficulty response performance index, and a third difficulty response performance index and recording them as comparison data; Acquire various matching data stored in a database, the matching data including a first difficulty response performance index matching value, a second difficulty response performance index matching value, and a third difficulty response performance index matching value; Compare the comparison data with the matching data one by one to determine the matching data that is most similar to the comparison data; The most similar matching data is obtained, and the verification result tolerance data stored in the database is obtained.

Images