Automatic construction method of aviation software knowledge data set based on LLM

Abstract

The invention discloses an automatic construction method of an aviation software knowledge data set based on LLM in the technical field of aviation airborne systems, and the method comprises the following steps: S1, generating question and answer pair data based on LLM role play, including role allocation, text segmentation and question and answer pair generation; s2, heuristically screening and filtering the question and answer pair data: adopting at least one large language model as an auditor, filtering low-quality questions firstly and then filtering low-quality answers so as to filter low-quality question and answer pairs; and S3, evaluating the quality and applicability of the question and answer pair data by adopting a mode of combining direct evaluation and indirect evaluation. According to the construction method, the data accuracy and the field adaptability are improved through semantic segmentation and multi-model fusion, the data quality is guaranteed through two-step heuristic filtering, and the data applicability is verified through multi-dimensional evaluation; the effect of efficiently constructing a high-quality and diversified aviation software knowledge data set is achieved.

Description

Technical Field

[0001] This invention relates to the field of airborne systems technology, and in particular to an automatic construction method for an aviation software knowledge dataset based on LLM. Background Technology

[0002] The use of software in airborne systems is growing rapidly, and aircraft operations are becoming increasingly reliant on software. This is especially true in recent years with the rise of the low-altitude economy, which has attracted numerous startups to the aviation software development industry. As the aviation software development industry flourishes, a large number of novice engineers join the field each year. However, aviation software development is characterized by its diverse types, complex processes, and broad range of disciplines, making it difficult for novice engineers to quickly master the relevant knowledge.

[0003] To address this, domestic domain experts have summarized their long-accumulated experience in aviation software development into a civil aircraft airborne software development system document. This document covers professional knowledge such as guidelines for conducting development activities at each stage of the aviation software development lifecycle and software practice standards, aiming to help practitioners alleviate practical difficulties. However, due to the complexity of the problems themselves, the role of this domain knowledge system document in practice is limited. Especially for novice engineers, finding information from numerous standards and guidelines to solve their problems requires a certain level of domain knowledge and considerable time. If a sufficient number of domain experts could provide hands-on guidance to novice engineers, they would effectively understand and master the domain knowledge and gradually accumulate their own domain experience. However, the practical difficulty lies in the limited number of domain experts with such capabilities, and the fact that their schedules are already packed. This ultimately creates a contradiction between the growing need of novice engineers for airborne software development knowledge and the shortage of domain experts and the difficulty in acquiring such knowledge.

[0004] The groundbreaking advancements of Large Language Models (LLM) in software engineering in recent years have offered new opportunities to alleviate the aforementioned contradictions. LLM plays a role in all stages of the software lifecycle, including requirements, design, development, and testing. As research deepens, researchers generally recognize that domain knowledge is essential for LLM to function effectively in specific domains. Building high-quality question-answering datasets based on existing engineering document libraries and targeting downstream question-answering tasks, and then using these datasets to train large language models, can yield a specialized aviation-domain model capable of responding to the concerns of novice engineers. This model can, to some extent, act as a domain expert, providing assistance to novice engineers through dialogue. However, extracting key knowledge points from domain knowledge documents and synthesizing them into datasets suitable for training LLM is a complex task. Manually constructing domain datasets is labor-intensive and struggles to guarantee dataset quality and diversity. Furthermore, data synthesis methods primarily rely on template construction and rule generation, which are insufficient for handling semantically complex tasks and large-scale data expansion. Summary of the Invention

[0005] This application provides an automatic construction method for aviation software knowledge datasets based on LLM, which solves the problems of large workload, inconsistent quality, and insufficient diversity in the manual construction of aviation software knowledge datasets in the prior art, and achieves the effect of efficiently constructing high-quality and diverse aviation software knowledge datasets.

[0006] This application provides an automatic construction method for an aviation software knowledge dataset based on LLM, including the following steps: S1: Question-answer pair data generated based on LLM role-playing, including: S1.1: Role Assignment: Based on LLM, define separate questioner and answerer roles, and design corresponding prompts for the two roles; S1.2: Text Segmentation: A semantic embedding model is used to perform semantic segmentation on aviation knowledge documents to obtain semantically coherent text blocks; S1.3: Question-answer pair generation: Questions are generated by the questioner and answers are generated by the respondent. Question-answer pair data are synthesized based on semantic consistency and confidence weighted fusion algorithm. S2: Heuristic filtering of the question-answer pair data includes: using at least one large language model as a moderator to first filter low-quality questions, then filter low-quality answers, thereby filtering low-quality question-answer pairs; S3: The quality and applicability of the question-and-answer pair data are evaluated using a combination of direct and indirect evaluation. The direct evaluation includes assessing the fidelity and diversity of the question-and-answer pair data. The indirect evaluation includes adjusting the LLM using the generated question-and-answer pair data and comparing the performance metrics of downstream tasks to evaluate the applicability of the question-and-answer pair data.

[0007] The beneficial effects of the above embodiments are as follows: the question-answer pairs constructed by this method possess both high quality and diversity, and the quality and applicability of the generated data are verified through a combination of direct and indirect evaluation, facilitating user decision-making. This construction method improves data accuracy and domain adaptability through semantic segmentation and multi-model fusion, ensures data quality through two-step heuristic filtering, and verifies data applicability through multi-dimensional evaluation; thus achieving the effect of efficiently constructing high-quality and diverse aviation software knowledge datasets.

[0008] Based on the above embodiments, this application can be further improved as follows: In one embodiment of this application, in step S1.1, the questioner role is played by the QwQ-32B model, and the answerer role is played by a set of at least two models selected from DeepSeek-R1-70B, Qwen2.5-32B, and ChatGLM-6B. Technical effect: Utilizing the domain adaptation capability of QwQ-32B to generate professional questions, and improving the professionalism, accuracy, and diversity of question-answer pairs through multi-model collaborative generation.

[0009] In one embodiment of this application, the semantic embedding model in S1.2 is the BGE large model. Technical effect: By leveraging the semantic understanding capabilities of BGE large, accurate semantic segmentation of aviation knowledge documents is achieved, avoiding information fragmentation and ensuring the semantic integrity and coherence of text blocks.

[0010] In one embodiment of this application, the semantic consistency and confidence-weighted fusion algorithm in S1.3 includes the following sub-steps: a) Construct the objective function: For the same problem, the model The probability function for the answer is: (1); in, For the observation vector, This is the optimal answer vector. This is a reliability parameter; a larger value indicates a stronger tendency for the output to be closer to the true answer. For vector similarity, This is a normalization constant; For the set of observation vectors for all models, the maximum likelihood estimate is: (2); in, To answer the question of the number of models, This is a multi-model reliability parameter vector; The overall goal is to maximize semantic consistency, and its objective function is: (3); in The regularization coefficient is . It is an L2 norm; b) and Parameter estimation and update iteration: Based on the objective function of equation (3), the alternating optimization algorithm is used to update... and The parameter values ​​are iterated until convergence: (4) (5); c) Text clustering, selecting the best cluster: By clustering the observed vectors, we obtain For each of the several clusters, calculate a score for that cluster: (6); in for The centroid of a cluster, For the model The confidence score, with a value of ; Select the best cluster This cluster represents the answer group with the highest semantic consistency. (7); d) Generate the final answer: From the best cluster In the middle, according to Select several high-weighted answers, extract key sentences / phrases, and merge them according to information coverage priority to form the final text.

[0011] Technical effect: The optimal semantic synthesis output by the semantic consistency and confidence weighted fusion algorithm effectively improves the accuracy and stability of question-answering data and reduces output bias.

[0012] In one embodiment of this application, the reviewer in S2 is DeepSeek-R1 LLM. The process of filtering low-quality questions is as follows: the text block generating the question-answer pair and the corresponding question-answer pair are input together into DeepSeek-R1 LLM. DeepSeek-R1 LLM scores each generated question based on pre-designed prompts, with scoring indicators covering four feature dimensions: engineering relevance, context fit, clarity of expression, and answerability. The process of filtering low-quality answers is as follows: the text block generating the question-answer pair and the corresponding question-answer pair are input together into DeepSeek-R1 LLM. DeepSeek-R1 LLM scores each generated answer based on pre-designed prompts, with scoring indicators covering four feature dimensions: accuracy, correctness, completeness, and feasibility. Technical effect: Utilizing the domain expertise of DeepSeek-R1, low-quality question-answer pairs are efficiently identified and filtered, reducing manual workload and significantly improving the overall quality of the question-answer pair dataset.

[0013] In one embodiment of this application, the fidelity assessment in the direct evaluation employs a BERT-based regression model, which uses a small sample of gold-standard data constructed by domain experts as the training set. Technical effect: The semantic understanding capability of the BERT model accurately quantifies the consistency between question-answer pairs and domain knowledge, reducing the cost of manual evaluation and ensuring the objectivity of the evaluation.

[0014] In one embodiment of this application, the diversity assessment in the direct evaluation includes three dimensions: the number of unique words, semantic similarity between word pairs, and the frequency of N-tuples between samples. Technical effect: This comprehensively measures the diversity of the dataset, ensuring that the data covers different aspects of domain knowledge and avoiding homogenization.

[0015] In one embodiment of this application, the downstream tasks in the indirect evaluation include document question answering and specification understanding, and the performance metrics are accuracy and F1 score. Technical effect: The effectiveness of the dataset is verified through the performance of downstream tasks, intuitively reflecting its value in improving aviation-specific LLM capabilities.

[0016] In one embodiment of this application, the prompt word consists of four parts: context, task, input, and output format. Constructing a prompt word engineering approach allows for the full utilization of LLM's generalization capabilities while better generating questions and answers that faithfully adhere to domain knowledge. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0018] Figure 1 This is a flowchart illustrating the automatic construction method of the aviation software knowledge dataset based on LLM in the embodiments of this application; Figure 2 This is a schematic diagram of the process for generating question-answer pair data in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the prompt words for generating the question in an embodiment of this application; Figure 4 This is a schematic diagram of the prompt words used to generate answers in an embodiment of this application; Figure 5 This is a schematic diagram illustrating the prompt words for LLM (Local Level Scale) in the embodiments of this application. Figure 6 This is a schematic diagram of LLM as a prompt word for answer scoring in an embodiment of this application. Detailed Implementation

[0019] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0020] Example: like Figure 1 As shown, an automatic construction method for an aviation software knowledge dataset based on LLM includes the following steps: S1: Generate question-answer pair data based on LLM role-playing, such as Figure 2 As shown, it includes: S1.1: Role Assignment: Based on LLM, define separate questioner and answerer roles, and design corresponding prompts for the two roles; To ensure the quality of the generated question-answer pairs, dedicated questioners and answerers were defined based on LLM. The questioner was represented by QwQ-32B, while the answerers were represented by multiple large models such as DeepSeek-R1-70B, Qwen2.5-32B, and ChatGLM-6B. Corresponding workflows and prompts were designed for each model to better obtain questions and answers faithful to domain knowledge, while also fully leveraging the generalization capabilities of LLM. Specific prompt templates are shown below. Figure 3 and Figure 4 As shown.

[0021] S1.2: Text Segmentation: A semantic embedding model is used to perform semantic segmentation on aviation knowledge documents to obtain semantically coherent text blocks; The input for synthetic question answering (CQA) data generation comes from text blocks of aviation software knowledge documents, which typically contain complex logical relationships and rich technical details. The text structure within these documents often includes extensive nesting, mutual references, and precise descriptions. Traditional equal-length segmentation strategies for processing such documents have significant drawbacks. Rigid equal-length segmentation easily disrupts the semantic coherence and integrity between sentence groups, leading to the loss of key contextual information. Fragmented semantics impair the quality of CQA data generation. To address these issues, this embodiment utilizes a semantic embedding model to deeply understand text semantics and dynamically adjust segmentation boundaries accordingly. The semantic embedding model can divide the document into text blocks according to the logical relationships and semantic relevance of the text, while ensuring that the length of the segmented text blocks is constrained within a specified range. This semantic-based text segmentation strategy effectively avoids the processing burden caused by excessively large text blocks or the information fragmentation caused by excessively small blocks, achieving a balance between ensuring semantic integrity and optimizing computational efficiency. This embodiment uses the BGE (BAAI General Embeddings) large model to achieve semantic segmentation of domain knowledge documents. This model performs excellently on various semantically related NLP tasks.

[0022] S1.3: Question-answer pair generation: Questions are generated by the questioner and answers are generated by the respondent. Question-answer pair data are synthesized based on semantic consistency and confidence weighted fusion algorithm. Question generation: Based on the prompts, the questioner raises several questions for each text block.

[0023] Multi-model answer generation: Using hybrid retrieval and progressive generation techniques, multiple large models use the same set of prompt words to retrieve highly relevant text blocks for each question. Based on the prompt words and highly relevant text blocks, an answer is generated for the question.

[0024] Multi-model response result fusion: The semantic consistency and confidence-weighted fusion algorithm includes the following sub-steps: a) Construct the objective function: For the same problem, the model The probability function for the answer is: (1); in, For the observation vector, This is the optimal answer vector. This is a reliability parameter; a larger value indicates a stronger tendency for the output to be closer to the true answer. For vector similarity, This is a normalization constant; For the set of observation vectors for all models, the maximum likelihood estimate is: (2); in, To answer the question of the number of models, This is a multi-model reliability parameter vector; The overall goal is to maximize semantic consistency, and its objective function is: (3); in The regularization coefficient is . It is an L2 norm; b) and Parameter estimation and update iteration: Based on the objective function of equation (3), the alternating optimization algorithm is used to update... and The parameter values ​​are iterated until convergence: (4) (5); c) Text clustering, selecting the best cluster: By clustering the observed vectors, we obtain For each of the several clusters, calculate a score for that cluster: (6); in for The centroid of a cluster, For the model The confidence score, with a value of ; Select the best cluster This cluster represents the answer group with the highest semantic consistency. (7); d) Generate the final answer: From the best cluster In the middle, according to Select several high-weighted answers, extract key sentences / phrases, and merge them according to information coverage priority to form the final text.

[0025] In the data generation stage, this embodiment adopts a large model answer fusion algorithm based on semantic consistency and confidence weighting. By constructing a weighted similarity objective function, the optimal semantic synthesis of multi-model outputs is achieved, which effectively improves the accuracy and stability of question and answer data. At the same time, a prompt word project consisting of four parts, context, task, input and output format, is constructed to better generate questions and answers that are faithful to domain knowledge while giving full play to the generalization ability of LLM.

[0026] S2: Heuristic filtering of question-answer pairs data, including: using at least one large language model as a moderator to first filter low-quality questions, then filter low-quality answers, thereby filtering low-quality question-answer pairs; The core objective of data filtering is to identify and filter out the low-quality portions of generated data pairs with the highest possible accuracy. This embodiment proposes a standards-based heuristic filtering strategy, using DeepSeek-R1 LLM as the moderator to filter low-quality question-answer pairs. The filtering process consists of two steps: first, filtering out question-answer pairs containing low-quality questions; and second, filtering out pairs containing low-quality answers.

[0027] A multidimensional evaluation algorithm based on a scoring function uses the model to score different dimensions for comprehensive evaluation.

[0028] Step 1: Filtering low-quality questions using standard guidance. A context-based filtering process was designed for the generated questions. In this process, the text block of the generated question is first used as the semantic context for evaluating the corresponding question. The text block and the question-answer pair are then input into the DeepSeek-R1 LLM scorer. The LLM scores based on the designed prompts (see...). Figure 5 Each generated question is scored, with scoring indicators covering features such as engineering relevance, contextual fit, clarity of expression, and answerability. Each dimension has its own independent scoring criteria.

[0029] Step Two: The filtering process for answers is similar to that for questions. The evaluation criteria for answers cover dimensions such as accuracy, correctness, completeness, and feasibility. LLM scores answers based on these dimensions. See the designed prompts below. Figure 6 In line with the evaluation and filtering process for questions, low-quality answers that do not meet the standards are filtered out.

[0030] In the data filtering stage, this embodiment constructs evaluation indicators from multiple dimensions to verify the effectiveness of the generated dataset and ensure the quality and diversity of the constructed dataset.

[0031] S3: The question-answer pair data is evaluated using a combination of direct and indirect evaluation. Direct evaluation includes assessing the fidelity and diversity of the question-answer pair data; indirect evaluation includes adjusting the LLM using the generated question-answer pair data and comparing the performance metrics of downstream tasks to evaluate the applicability of the question-answer pair data.

[0032] After the data filtering process is completed, a combination of direct and indirect evaluation methods is used to assess the quality and applicability of the generated question-and-answer set to the dataset.

[0033] 1) Direct assessment: Direct evaluation includes assessments of the dataset’s fidelity and diversity (i.e., quality).

[0034] In terms of dataset fidelity assessment, to accurately reflect that the source document content has not introduced the LLM illusion, this embodiment introduces a gold standard based on expert experience to assist in model evaluation.

[0035] The model evaluation method uses a small-sample gold standard based on domain expert experience to train a supervised model to quantify the fidelity score of the generated question-answer pairs. The specific process is as follows: (1): Construction of small-sample gold standard. Domain experts create a corresponding number of related question-answer pairs for each topic according to the topic proportion in the domain knowledge document. The creation of question-answer pairs is based on the domain experts' understanding of the domain knowledge document and their actual engineering experience, combined with evaluation criteria to ensure the quality of the construction results; (2): Alignment Evaluation Process. Using the seed dataset as the training set, a regression model based on BERT is trained to predict the fidelity scores of question-answer pairs in the generated dataset. The trained model is then applied to the generated dataset, and the score distribution of the question-answer pairs is analyzed. If a question-answer pair's score is close to or exceeds the mean of the seed dataset, it indicates that the question-answer pair is consistent with the expert-constructed sample in terms of content, thus verifying its fidelity. This method significantly reduces the reliance on large-scale manual review while ensuring the objectivity and scalability of the evaluation.

[0036] In terms of diversity assessment, the diversity of generated data has a significant impact on the generalization ability of downstream tasks. Therefore, this patent analyzes the data from two levels: lexical diversity and sample relevance, and designs three indicators: the number of unique words, semantic similarity between word pairs, and frequency of N-tuples between samples. These indicators can reflect the diversity of the generated question-and-answer pairs in terms of vocabulary, semantics, and expression.

[0037] Lexical diversity reflects the richness of vocabulary usage in a dataset and is an important indicator for measuring surface-level differences in datasets. This embodiment uses the number of unique words as a metric, calculating the number of non-repeating words in both the generated dataset and the domain knowledge text blocks from which they originate. We compare the unique word lists of the two datasets using word embeddings to obtain the similarity / difference rate in word usage between the generated data and the data source.

[0038] Sample relevance: Semantic similarity between word pairs measures the semantic differences between generated samples. The number of unique words is used to calculate the semantic similarity between any question-answer pairs based on the embedding vectors, and kernel density estimation (KDE) is used to model the similarity distribution. The density structure of the similarity distribution can reveal the degree of sample clustering in the semantic space, thus reflecting the level of semantic diversity of the data. The inter-sample N-tuple frequency (INGF) is used to assess the repetition of generated samples in local expression segments. By statistically analyzing the shared N-tuples between question-answer pairs, the diversity of data in language expression structure is quantified. A lower INF score represents higher expression differences.

[0039] 2) Indirect assessment: The performance improvements of downstream tasks are used as an indirect basis for evaluating the applicability of the synthetic dataset. The specific process is as follows.

[0040] Step 1: Fine-tune the LLM using a synthetic dataset. The constructed question-answering pair dataset is used to perform domain-specific fine-tuning of the general LLM. Fine-tuning maps the general LLM to a domain-specific knowledge space, enabling it to grasp domain knowledge and semantic features. The fine-tuning process does not introduce external data, relying solely on the synthetic dataset to highlight the direct contribution of the dataset to the model's capabilities.

[0041] Step 2: Downstream Task Performance Comparison. After model fine-tuning, select downstream tasks related to the target domain (such as document question answering and specification understanding) for testing, and record the model performance metrics (such as accuracy, F1 score, etc.) before and after fine-tuning. Through comparative analysis, evaluate the actual value of the synthetic dataset in improving model performance.

[0042] During the data evaluation phase, this embodiment assesses the quality and applicability of the filtered data from both direct and indirect perspectives to facilitate user decision-making.

[0043] An automatic construction device for an aviation software knowledge dataset based on LLM, employing the aforementioned automatic construction method, includes: The data generation module is used to generate question-and-answer pair data based on LLM role-playing, including: Role assignment: Based on LLM, separate questioner and answerer roles are defined, and corresponding prompt words are designed for the two roles; Text segmentation: Semantic embedding model is used to perform semantic segmentation on aviation knowledge documents to obtain semantically coherent text blocks; Question-and-answer pair generation: Questions are generated by the questioner role, and answers are generated by the answerer role. Question-and-answer pair data is synthesized based on semantic consistency and confidence weighted fusion algorithm. The data filtering module is used to heuristically filter question-answer pair data, including: using at least one large language model as a reviewer to first filter low-quality questions, then filter low-quality answers, thereby filtering low-quality question-answer pairs; The quality assessment module is used to evaluate the quality and applicability of question-answer pair data using a combination of direct and indirect assessment methods. Direct assessment includes evaluating the fidelity and diversity of the question-answer pair data; indirect assessment includes: adjusting the LLM using the generated question-answer pair data and comparing the performance metrics of downstream tasks to evaluate the applicability of the question-answer pair data.

[0044] The functions of each module correspond to the content of its corresponding method implementation, and will not be elaborated here.

[0045] The technical solutions described in the embodiments of this application above have at least the following technical effects or advantages: (1) This automatic construction method can ensure the quality and diversity of dataset construction through three stages: data generation, data filtering and quality assessment, and effectively solve the problems of large workload, uneven quality and insufficient diversity of manual construction.

[0046] (2) In the data generation stage, this automatic construction method adopts a large model answer fusion algorithm based on semantic consistency and confidence weighting, combined with the corresponding workflow and prompt words, to give full play to the generalization ability of LLM and better obtain questions and answers that are faithful to aviation software knowledge; (3) This automatic construction method proposes a standard-based heuristic filtering strategy, using LLM to filter low-quality question-answer pair data and improve the accuracy of the data; at the same time, it constructs an evaluation index system from multiple dimensions, and verifies the quality and applicability of the generated data through a combination of direct and indirect evaluation.

[0047] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. An automatic construction method for an aviation software knowledge dataset based on LLM, characterized in that, Includes the following steps: S1: Question-answer pair data generated based on LLM role-playing, including: S1.1: Role Assignment: Based on LLM, define separate questioner and answerer roles, and design corresponding prompts for the two roles; S1.2: Text Segmentation: A semantic embedding model is used to perform semantic segmentation on aviation knowledge documents to obtain semantically coherent text blocks; S1.3: Question-answer pair generation: Questions are generated by the questioner and answers are generated by the respondent. Question-answer pair data are synthesized based on semantic consistency and confidence weighted fusion algorithm. S2: Heuristic filtering of the question-answer pair data includes: using at least one large language model as a moderator to first filter low-quality questions, then filter low-quality answers, thereby filtering low-quality question-answer pairs; S3: The quality and applicability of the question-and-answer pair data are evaluated using a combination of direct and indirect evaluation. The direct evaluation includes assessing the fidelity and diversity of the question-and-answer pair data. The indirect evaluation includes adjusting the LLM using the generated question-and-answer pair data and comparing the performance metrics of downstream tasks to evaluate the applicability of the question-and-answer pair data.

2. The automatic construction method according to claim 1, characterized in that: In S1.1, the questioner role is played by the QwQ-32B model, and the respondent role is played by a set of at least two models from DeepSeek-R1-70B, Qwen2.5-32B, and ChatGLM-6B.

3. The automatic construction method according to claim 1, characterized in that: The semantic embedding model in S1.2 is the BGE large model.

4. The automatic construction method according to claim 1, characterized in that: The semantic consistency and confidence-weighted fusion algorithm in S1.3 includes the following sub-steps: a) Construct the objective function: For the same problem, the model The probability function for the answer is: （1）； in, For the observation vector, This is the optimal answer vector. This is a reliability parameter; a larger value indicates a stronger tendency for the output to be closer to the true answer. For vector similarity, This is the normalization constant; For the set of observation vectors for all models, the maximum likelihood estimate is: （2）； in, To answer the question of the number of models, This is a multi-model reliability parameter vector; The overall goal is to maximize semantic consistency, and its objective function is: （3）； in The regularization coefficient is . It is an L2 norm; b) and Parameter estimation and update iteration: Based on the objective function of equation (3), the alternating optimization algorithm is used to update... and The parameter values ​​are iterated until convergence: （4） （5）； c) Text clustering, selecting the best cluster: By clustering the observed vectors, we obtain For each of the several clusters, calculate a score for that cluster: （6）； in for The centroid of a cluster, For the model The confidence score, with a value of ; Select the best cluster This cluster represents the answer group with the highest semantic consistency. （7）； d) Generate the final answer: From the best cluster In the middle, according to Select several high-weighted answers, extract key sentences / phrases, and merge them according to information coverage priority to form the final text.

5. The automatic construction method according to claim 1, characterized in that: The reviewer in S2 is DeepSeek-R1 LLM. The process of filtering low-quality questions is as follows: The text block of the generated question-answer pair and the corresponding question-answer pair are input into DeepSeek-R1 LLM. DeepSeek-R1 LLM scores each generated question based on pre-designed prompts. The scoring indicators cover four feature dimensions: engineering relevance, context fit, clarity of expression, and answerability. The process of filtering low-quality answers is as follows: The text block of the generated question-answer pair and the corresponding question-answer pair are input into DeepSeek-R1 LLM. DeepSeek-R1 LLM scores each generated answer based on pre-designed prompts. The scoring indicators cover four feature dimensions: accuracy, correctness, completeness, and feasibility.

6. The automatic construction method according to claim 1, characterized in that: The loyalty assessment in the direct evaluation uses a BERT-based regression model, which is trained on a small sample of gold standard data constructed by domain experts.

7. The automatic construction method according to claim 6, characterized in that: The diversity assessment in the direct evaluation includes indicators in three dimensions: the number of unique words, semantic similarity between word pairs, and frequency of N-tuples between samples.

8. The automatic construction method according to claim 1, characterized in that: The downstream tasks in the indirect evaluation include document question answering and specification understanding, and the performance metrics are accuracy and F1 score.

9. The automatic construction method according to claim 1, characterized in that: The prompt word consists of four parts: context, task, input, and output format.

10. An automatic construction device for an aviation software knowledge dataset based on LLM, characterized in that, The automatic construction method as described in any one of claims 1-9 includes: The data generation module is used to generate question-and-answer pair data based on LLM role-playing, including: Role assignment: Based on LLM, separate questioner and answerer roles are defined, and corresponding prompt words are designed for the two roles; Text segmentation: Semantic embedding model is used to perform semantic segmentation on aviation knowledge documents to obtain semantically coherent text blocks; Question-and-answer pair generation: Questions are generated by the questioner role, and answers are generated by the answerer role. Question-and-answer pair data is synthesized based on semantic consistency and confidence weighted fusion algorithm. The data filtering module is used to heuristically filter the question-answer pair data, including: using at least one large language model as a reviewer to first filter low-quality questions, then filter low-quality answers, thereby filtering low-quality question-answer pairs; The quality assessment module is used to evaluate the quality and applicability of the question-and-answer pair data using a combination of direct and indirect assessment. The direct assessment includes evaluating the fidelity and diversity of the question-and-answer pair data; the indirect assessment includes: adjusting the LLM using the generated question-and-answer pair data and comparing the performance metrics of downstream tasks to evaluate the applicability of the question-and-answer pair data.

Images