Efficient evaluation method for large model of aerospace control software

Abstract

The invention provides an efficient evaluation method for a large model of aerospace control software, which comprises the following steps of: extracting professional corpora from a model document of the aerospace control software to form a professional corpus; embedding each corpus into a semantic vector, classifying the semantic vectors, and retaining the corpus closest to a clustering center in each class to form a simplified corpus; taking each corpus in the simplified corpus as the input of the general large model, and generating a multi-type evaluation test question set; the general large model is a non-to-be-evaluated large model; screening the test question set by utilizing cross validation of a plurality of general large models to form an effective test question set; the test question set comprises test questions and answers; inputting the test questions in the effective test question set into the to-be-evaluated large model, and evaluating the test questions to obtain scores of the to-be-evaluated large model for each type of test questions; and carrying out weighted average on the score of each type of test questions to obtain a final evaluation score of the to-be-evaluated large model.

Description

Technical Field

[0001] This invention belongs to the field of aerospace control software, specifically involving an efficient evaluation method for large-scale aerospace control software models. Background Technology

[0002] In the development of large-scale models for aerospace control software, large-scale model evaluation plays a crucial role in assessing domain capabilities and providing insights for future improvements. However, common domain evaluation methods for large-scale models rely on experts to construct evaluation question sets, requiring significant manpower and time. Furthermore, they require internationally advanced closed-source large-scale models (such as ChatGPT-4) to evaluate the quality of the large-scale models' responses, which is not feasible in the aerospace field. Summary of the Invention

[0003] The technical problem solved by this invention is to propose an efficient evaluation method for large models of aerospace control software, which significantly reduces the manpower and time costs of large model evaluation while ensuring evaluation quality.

[0004] The solution provided by this invention is: an efficient evaluation method for large-scale models of aerospace control software, characterized by comprising:

[0005] Specialized corpus was formed by extracting specialized language from the model documents of aerospace control software.

[0006] Each piece of text is embedded as a semantic vector, the semantic vectors are classified, and the text closest to the cluster center in each category is retained to form a concise corpus.

[0007] Each piece of text in the concise corpus is used as input to a general large model to generate a set of evaluation questions with multiple contents and types; the general large model is not the large model to be evaluated.

[0008] The test question set is filtered by cross-validation using multiple general large models to form an effective test question set; the test question set includes test questions and answers.

[0009] The questions from the valid question set are input into the large model to be evaluated, and the questions are evaluated to obtain the score of the large model for each type of question.

[0010] The scores for each type of question are weighted and averaged to obtain the final evaluation score of the large model to be evaluated.

[0011] Preferably, the model documentation includes user requirements, mission specifications, communication protocols, software requirements specifications, software overview design, software detailed design, unit test and integration test reports for the aerospace control software.

[0012] Preferably, the professional corpus includes three types of heterogeneous data: formulas, tables, and text. The formula category includes logic and operational formulas for initialization, attitude determination, target attitude calculation, spacecraft control and propulsion, and fault detection algorithms. The table category includes communication protocol data, requirement traceability matrix, data structure design, unit test and integration test table data. The text category includes control software functional performance requirements and design, reliability and safety requirements and design, software function interface design, and test summary textual descriptions.

[0013] Preferably, the multiple types include multiple choice questions, true / false questions, and open-ended questions.

[0014] Preferably, the evaluation of question-and-answer questions first compares the answers generated by the large model to be evaluated with the corresponding answers in the valid question set, and calculates their text similarity score and semantic similarity score. If there are multiple answers to the question-and-answer questions in the valid question set, referred to as multiple standard answers, the answers generated by the large model to be evaluated are ranked and compared with the corresponding answers in the valid question set. If the generated answer is ranked higher than a certain standard answer, it is scored as 1 point; otherwise, it is scored as 0 points. Finally, the text similarity score, semantic similarity score, and ranking score are weighted and averaged to obtain the final answer score.

[0015] Preferably, the text similarity score, semantic similarity score, and ranking score are weighted and averaged; let the text similarity score be S. t The semantic similarity score is S s The sorting score is S. o The weighting coefficients are α, β, and γ, and the final question-and-answer score S is... q Represented as: S q =α·S t +β·S s +γ·S o The text similarity score S t The weighting coefficient α is suggested to have a range of [0.2, 0.3], and the semantic similarity score is S. s The weighting coefficient β is suggested to have a value range of [0.4, 0.5], and the ranking score S o The weighting coefficient γ is suggested to have a range of [0.2, 0.4], and α + β + γ = 1. Using the above expression and suggested weight range, the importance of each scoring indicator can be flexibly adjusted. For example, if semantic understanding is more critical, the value of β can be increased; if text matching is more important, the weight of γ can be appropriately increased.

[0016] Preferably, a weighted average is calculated for the scores of each type of question. Let S be the score for the multiple-choice questions. c The score for the true / false questions is S. j The score for the short answer question is S. qThe corresponding weighting coefficients are α', β', and γ', respectively. Therefore, the final score calculation formula for this type of question is: S type =α'·S c +β'·S j +γ'·S q The suggested value range for α' is [0.2, 0.3] for multiple-choice questions, [0.1, 0.2] for true / false questions, and [0.5, 0.7] for short-answer questions, with α' + β' + γ' = 1. The weighting coefficients can be adjusted according to actual needs to reflect the importance of different question types.

[0017] Preferably, the test questions are subcategorized according to the data format, including questions containing formulas, questions containing tables, and questions containing text. Each subcategory contains the three types of questions described in claim 7. A weighted average is then calculated for the scores of each subcategory. Let n be the number of questions containing formulas. f The number of questions containing tables is n. t The number of test questions containing text is n. o The total number of questions is N = n f +n t +n o Questions containing formulas are scored as S. type_f The number of test questions containing tables is S. type_t The number of test questions containing text is S. type_o The final evaluation score is calculated using the following formula: Among them, the questions containing formulas correspond to The suggested value range is [0.4, 0.6], including the questions in the table. The suggested value range is [0.3, 0.5], corresponding to questions containing text. The recommended value range is [0.1, 0.3], and The weighting coefficients can be adjusted according to actual needs to reflect the importance of different question types.

[0018] A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of an efficient evaluation method for a large model of aerospace control software.

[0019] A computer software product includes: a processor and a storage device;

[0020] Storage device for storing one or more programs;

[0021] When the one or more programs are executed by one or more processors, the one or more processors implement the efficient evaluation method for large-scale aerospace control software models.

[0022] The advantages of this invention compared to the prior art are:

[0023] This invention provides a highly efficient evaluation method for large-scale models in the aerospace control software field, significantly reducing the manpower and time costs of domain evaluation. As the domain corpus expands, this method can flexibly extend the evaluation scope without redesigning the evaluation framework. The evaluation results not only allow for in-depth analysis of the large model's knowledge mastery within the aerospace field but also provide data support for the continuous optimization of the large model. Therefore, this method offers a low-cost domain evaluation approach that can be applied to question-and-answer domains such as requirements, design, and programming for large-scale aerospace control software models.

[0024] This invention proposes a novel evaluation method for large-scale models of aerospace control software. By automatically generating evaluation questions and utilizing cross-validation of the large model to screen the questions, it significantly reduces manual intervention and improves the objectivity and efficiency of the evaluation. The proposed method uses cross-validation of the large model to ensure the quality of question generation and employs multi-dimensional evaluation metrics to effectively assess the model's responses. This approach not only leverages the large model's generation capabilities to improve the efficiency of large-scale model evaluation but also ensures the quality of the evaluation process.

[0025] The evaluation method of this invention can effectively reduce the workload of domain experts and can be flexibly expanded to adapt to the ever-growing aerospace corpus, thereby ensuring the efficiency and stability of the evaluation process. This method does not rely on internationally advanced closed-source large models and is fully applicable to the aerospace control software field, and can be directly used for the continuous optimization of domain-specific large models. Detailed Implementation

[0026] The present invention will be further described below with reference to the embodiments.

[0027] This method consists of two parts: generating a set of evaluation questions and evaluating the performance of a large model.

[0028] I. Question Generation

[0029] 1. Extract relevant professional corpora from the model documents of aerospace control software. Model documents include user requirements, mission specifications, communication protocols, software requirement specifications, software overview design, software detailed design, unit test reports, and integration test reports. The professional corpus includes three types of heterogeneous data: formulas, tables, and text. Formulas include logical and computational formulas for algorithms such as initialization, attitude determination, target attitude calculation, spacecraft control and propulsion, and fault detection. Tables include communication protocol data, requirement traceability matrices, data structure designs, unit test reports, and integration test reports. Text includes textual descriptions of control software functional performance requirements and designs, reliability and safety requirements and designs, software function interface designs, and test summaries. The extraction method is as follows: traverse the document page by page, input a general multimodal large model (such as RolmOCR-7B) to obtain Markdown text format, then use regular expressions to extract tables, formulas, and text from the document, remove duplicate descriptions, and save the extracted data to the corpus.

[0030] 2. Corpus selection based on a large model. Each piece of text is embedded as a semantic vector, and then these semantic vectors are classified using the k-means clustering algorithm. Finally, the text closest to the cluster center in each class is retained to form a concise corpus.

[0031] 3. Use prompt engineering techniques to generate various types of assessment questions. In this step, the large language model will automatically generate multiple-choice, true / false, and open-ended questions based on specific prompts. The large language model here is a general model, such as the Qwen2-72B model.

[0032] Example of generating prompts for multiple-choice questions:

[0033] "Based on the following description of aerospace control software, generate a related multiple-choice question and provide four options, one of which is the correct answer: '[Description of aerospace control software]'."

[0034] Example of generating prompts for true / false questions:

[0035] "Based on the following description of the aerospace control software, generate a true / false question. The question should accurately reflect the description and indicate whether it is true or false: '[Description of the aerospace control software]'."

[0036] Example of prompts for generating questions and answers:

[0037] "Based on the following description of aerospace control software, generate a question and answer, ensuring that the question guides the respondent to provide a detailed explanation of the description: '[Description of aerospace control software]'."

[0038] 4. Cross-validation to filter the question set. Multiple large language models (e.g., Qwen2-7B, ChatGLM4-9B, and Baichuan2-7B) are used for cross-validation to further filter the question set. Specifically, each large model answers and votes on the generated questions, retaining those that receive support from multiple models. Questions with low votes or zero votes are deleted.

[0039] 5. For open-ended questions, after cross-validation, prompting engineering techniques are used to expand a single standard answer into multiple standard answers with different descriptions but the same semantic meaning, for subsequent evaluation. Example of answer expansion prompts:

[0040] "Based on the answers to the following aerospace control software questions, expand to generate multiple answers with different descriptions but the same semantics, and ensure that these answers are all correct: '[Aerospace control software question]' [Aerospace control software answer]'."

[0041] 6. For the three heterogeneous data types of formulas, tables and text in the corpus, multiple choice, true / false and open-ended questions will be generated according to the above process 3-5.

[0042] II. Test Item Evaluation

[0043] 1. After the test question set is generated, it will be input into the large-scale model of the aerospace control software to be evaluated. After the large-scale model generates answers, its domain performance is evaluated by analyzing the consistency between the generated answers and the answers in the test question set. For ease of subsequent description, it is assumed that the number of formula-based test questions is n. f The number of table-type questions is n. t The number of text-based questions is n. o The total number of questions is N = n f +n t +n o .

[0044] 2. Taking formula-based questions as an example, steps 3-5 are used to calculate the score S for formula-based questions. type_f Score for table-type questions type_t Score for text-based test questions type_o The calculation process is the same. Assume the number of multiple-choice, true / false, and open-ended questions in the formula-based test are n respectively. c n j n q And n f =n c +n j +n q .

[0045] 3. For multiple-choice and true / false questions, evaluate the accuracy rate. Assume the number of correctly answered multiple-choice and true / false questions are n respectively. cT and n jT,but:

[0046]

[0047] Among them, S c and S j The correct scores for multiple-choice and true / false questions are respectively.

[0048] 4. Compared to multiple-choice and true / false questions, evaluating open-ended questions is more complex and requires multi-dimensional evaluation metrics. Specifically, firstly, the answers generated by a large model are compared with the standard answers to calculate their text similarity score (Rogue-L) and semantic similarity score (BertScore). Secondly, a ranking model is introduced to rank and compare the answers generated by the large model with multiple standard answers. If the generated answer ranks higher than a certain standard answer, it receives 1 point; otherwise, it receives 0 points. Finally, the text similarity score, semantic similarity score, and ranking score are weighted and averaged to obtain the final answer score.

[0049] Rouge-L scoring formula:

[0050]

[0051] Among them, A k Let R represent the k-th reference answer set, S represent the sentence in the reference answer, w represent the words in the sentence, and Count(w) represent the number of matching words between the candidate answer and the reference answer.

[0052] BertScore scoring formula:

[0053]

[0054] Where P and Q represent the word embedding vector sequences of the candidate answer and the reference answer, respectively, and CosSim(P i Q j ) represents the cosine similarity between word embedding vectors.

[0055] Sorting score formula:

[0056]

[0057] Among them, C k Let be the candidate answer for the k-th question, and let 1 be the indicator function. If the candidate answer is ranked higher than any of the standard answers, it is scored as 1 point; otherwise, it is scored as 0 points.

[0058] Let the weighting coefficients be α, β, and γ, then the answer score S for the question-and-answer question is... q It can be represented as: S q =α·S t +β·S s+γ·S o The text similarity score S t The weighting coefficient α is suggested to have a range of [0.2, 0.3], and the semantic similarity score is S. s The weighting coefficient β is suggested to have a value range of [0.4, 0.5], and the ranking score S o The weighting coefficient γ is recommended to have a range of [0.2, 0.4], and α + β + γ = 1.

[0059] 5. Weighted Calculation of Scores for a Certain Type of Question. Let the weighting coefficients for multiple-choice, true / false, and short-answer questions be α', β', and γ', respectively. Then, the final score calculation formula for this type of question is: S type_f =α'·S c +β'·S j +γ'·S q The suggested value range for α' for multiple-choice questions is [0.2, 0.3], for β' for true / false questions is [0.1, 0.2], and for γ' for open-ended questions is [0.5, 0.7], with α' + β' + γ' = 1.

[0060] 6. Weighted Calculation of the Final Score of the Domain Model. After evaluating all question types, the final step is to calculate a weighted average of the scores for the large model to obtain its final evaluation score in the field of aerospace control. The scoring formula is as follows: Among them, the questions containing formulas correspond to The suggested value range is [0.4, 0.6], including the questions in the table. The suggested value range is [0.3, 0.5], corresponding to questions containing text. The recommended value range is [0.1, 0.3], and

[0061] The present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the efficient evaluation method for a large model of aerospace control software.

[0062] The present invention further provides a computer software product, including: a processor and a storage device;

[0063] Storage device for storing one or more programs;

[0064] When the one or more programs are executed by one or more processors, the one or more processors implement the efficient evaluation method for large-scale aerospace control software models.

[0065] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

[0066] The parts of this invention not described in detail are common knowledge to those skilled in the art.

Claims

1. A highly efficient evaluation method for large-scale models of aerospace control software, characterized in that... include: Specialized corpus was formed by extracting specialized language from the model documents of aerospace control software. Each piece of text is embedded as a semantic vector, the semantic vectors are classified, and the text closest to the cluster center in each category is retained to form a concise corpus. Each piece of text in the concise corpus is used as input to a general large model to generate a set of evaluation questions with multiple contents and types; the general large model is not the large model to be evaluated. The test question set is filtered by cross-validation using multiple general large models to form an effective test question set; the test question set includes test questions and answers. The questions from the valid question set are input into the large model to be evaluated, and the questions are evaluated to obtain the score of the large model for each type of question. The scores for each type of question are weighted and averaged to obtain the final evaluation score of the large model to be evaluated.

2. The method according to claim 1, characterized in that: The aforementioned model documentation includes user requirements, mission specifications, communication protocols, software requirements specifications, software overview design, software detailed design, unit test reports, and integration test reports for the aerospace control software.

3. The method according to claim 1, characterized in that: The aforementioned professional corpus includes three types of heterogeneous data: formulas, tables, and text. The formula category includes logic and operational formulas for initialization, attitude determination, target attitude calculation, spacecraft control and propulsion, and fault detection algorithms. The table category includes communication protocol data, requirement traceability matrix, data structure design, unit test and integration test table data. The text category includes control software functional performance requirements and design, reliability and safety requirements and design, software function interface design, and test summary textual descriptions.

4. The method according to claim 1, characterized in that: The various question types include multiple choice, true / false, and short answer questions.

5. The method according to claim 4, characterized in that: The evaluation of question-and-answer questions first compares the answers generated by the large model to be evaluated with the corresponding answers in the valid question set, calculating their text similarity score and semantic similarity score. If there are multiple corresponding answers in the valid question set, referred to as multiple standard answers, the answers generated by the large model to be evaluated are ranked and compared with the corresponding answers in the valid question set. If the generated answer is ranked higher than a certain standard answer, it is scored as 1 point; otherwise, it is scored as 0 points. Finally, the text similarity score, semantic similarity score, and ranking score are weighted and averaged to obtain the final answer score.

6. The method according to claim 5, characterized in that: The text similarity score, semantic similarity score, and ranking score are weighted and averaged; let the text similarity score be S. t The semantic similarity score is S s The sorting score is S. o The weighting coefficients are α, β, and γ, and the final question-and-answer score S is... q Represented as: S q =α·S t +β·S s +γ·S o The text similarity score S t The weighting coefficient α is suggested to have a range of [0.2, 0.3], and the semantic similarity score is S. s The weighting coefficient β is suggested to have a value range of [0.4, 0.5], and the ranking score S o The weighting coefficient γ is recommended to have a range of [0.2, 0.4], and α + β + γ = 1.

7. The method according to claim 4, characterized in that: The scores for each type of question are weighted and averaged. Let S be the score for the multiple-choice questions. c The score for the true / false questions is S. j The score for the short answer question is S. q The corresponding weighting coefficients are α', β', and γ', respectively. Therefore, the final score calculation formula for this type of question is: S type =α'·S c +β'·S j +γ'·S q Among them, the suggested value range for α' for multiple-choice questions is [0.2, 0.3], the suggested value range for β' for true / false questions is [0.1, 0.2], and the suggested value range for γ' for open-ended questions is [0.5, 0.7], and α' + β' + γ' = 1.

8. The method according to claim 7, characterized in that: Based on the data format, the questions are divided into subcategories: questions containing formulas, questions containing tables, and questions containing text. Each subcategory contains three types of questions: multiple choice, true / false, and open-ended questions. A weighted average is calculated for the scores of each subcategory. Let n be the number of questions containing formulas. f The number of questions containing tables is n. t The number of test questions containing text is n. o The total number of questions is N = n f +n t +n o Questions containing formulas are scored as S. type_f The number of test questions containing tables is S. type_t The number of test questions containing text is S. type_o The final evaluation score is calculated using the following formula: Among them, the questions containing formulas correspond to The suggested value range is [0.4, 0.6], including the questions in the table. The suggested value range is [0.3, 0.5], corresponding to questions containing text. The recommended value range is [0.1, 0.3], and 9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the efficient evaluation method for a large model of aerospace control software as described in any one of claims 1-8.

10. A computer software product, characterized in that: include: Processors and storage devices; Storage device for storing one or more programs; When the one or more programs are executed by one or more processors, the one or more processors implement the efficient evaluation method for large-scale aerospace control software models as described in any one of claims 1-8.

Images