Frequency track declaration coordinated question-answering system and method based on large model

Abstract

The invention relates to the field of satellite frequency orbit resources, in particular to a frequency orbit declaration coordinated question-answering system and method based on a large model. The system comprises a data processing module used for obtaining and preprocessing original data and constructing a pre-training data set and a question and answer pair data set; the model training optimization module is used for taking the pre-training data set as an original data set, taking the question and answer pair data set as a target data set, and generating a training data set through topic distribution modeling and text importance evaluation in combination with a sampling screening strategy; the big model pre-training module is also used for optimizing the training process of the big model based on data screening of training loss and a dynamic batch updating strategy so as to complete big model pre-training; and the model fine tuning deployment module is used for performing fine tuning on the pre-trained large model by adopting the question and answer pair data set, and deploying the fine-tuned large model to realize a frequency orbit declaration coordinated question and answer function. According to the invention, the efficiency and accuracy of frequency track declaration coordination can be improved.

Description

Technical Field

[0001] This application relates to the field of satellite frequency and orbit resources, and in particular to a frequency and orbit application coordination question-and-answer system and method based on a large model. Background Technology

[0002] Artificial Earth satellites are unmanned spacecraft manufactured and launched into Earth orbit by humans. The radio frequencies and satellite orbits required for satellite launches are non-renewable natural resources with enormous economic and practical value. In recent years, countries have been accelerating their efforts to secure the limited total number of frequency and orbit resources. my country's satellite constellation lacks frequency and orbit resources that are interceptable, have high priority, and are easy to use, putting it at a disadvantage in international rule-making and cooperation. Therefore, it is urgent to actively apply for and reserve frequency and orbit resources.

[0003] Existing methods primarily aim to expedite the application and coordination process from an auxiliary perspective. Each application relies on traditional information retrieval methods, resulting in low efficiency and unreliable accuracy. Large-scale model-based question-answering systems, as an emerging artificial intelligence technology, can understand users' information needs expressed in natural language, offering an effective solution to these problems. However, large-scale models for specific domains require high-quality question-answering data, yet the satellite communication field currently lacks such research. Furthermore, training large-scale models requires extensive computation on massive datasets over extended periods. Existing large-scale model training optimization algorithms focus only on selecting training data or optimizing the training process, optimizing from a single direction. Therefore, researching a method for efficiently training large-scale models is also crucial. Summary of the Invention

[0004] The purpose of this application is to overcome the above-mentioned deficiencies of the prior art, thereby providing a frequency track declaration coordination question and answer system and method based on a large model.

[0005] To address the aforementioned technical problems, the technical solution provided in this application offers a frequency track declaration coordination question-and-answer system based on a large model, comprising: The data processing module is used to acquire raw data and preprocess it to build a pre-training dataset and a question-answering pair dataset. The model training optimization module uses the pre-training dataset as the original dataset and the question-answer pair dataset as the target dataset. It generates the training dataset through topic distribution modeling and text importance evaluation, combined with a sampling and filtering strategy. It also optimizes the training process of large models based on data filtering and dynamic batch update strategies using training loss, thereby completing the pre-training of large models. The model fine-tuning and deployment module is used to fine-tune the pre-trained large model using a question-and-answer pair dataset, and then deploy the fine-tuned large model to realize the frequency track declaration coordination question-and-answer function.

[0006] As an improvement to the aforementioned system, the raw data includes: a series of radio rules documents.

[0007] As an improvement to the aforementioned system, the data processing module includes a data acquisition and format conversion unit, a data structuring processing unit, a data cleaning and optimization unit, and a question-answer pair generation unit. Specifically, the data acquisition and format conversion unit extracts and stores the text and tabular data from the original data; the data structuring processing unit converts the text and tabular data of the original data into independent data; the data cleaning and optimization unit cleans the converted data through rule filtering, data deduplication, and text segmentation steps to obtain the pre-trained dataset; and the question-answer pair generation unit, based on frequency-track reporting and coordination of high-frequency query question types, extracts corresponding answers from the pre-trained dataset using automated scripts to generate the question-answer pair dataset.

[0008] As an improvement to the above system, the data structuring processing unit is used to divide text data based on paragraphs, converting each paragraph of text data into independent data; it is used to concatenate the table header information of the row content of the table data with the specific content of the row to which the header information belongs, thereby converting the row content table into independent data; and it is used to concatenate the table header information of the column content of the table data with the content of the column to which the header information belongs, thereby converting the column content table into independent data.

[0009] As an improvement to the above system, the frequency track declaration coordination high-frequency query problem types include: business consultation, limit query, and footnote retrieval.

[0010] As an improvement to the above system, the model training optimization module includes a data selection submodule and a training optimization submodule; wherein, the data selection submodule is used to use the pre-training dataset as the original dataset. The question-answer pair dataset is used as the target dataset. By modeling topic distribution and evaluating text importance, a topic model is used to analyze the original dataset. With the target dataset Topic mining was performed to obtain the original datasets. and target dataset Topic distribution; used to calculate the original dataset The similarity between the topic distribution of the Chinese text t and the target dataset is used as the basis for the original dataset. Importance weight of Chinese text t Used to calculate importance scores in conjunction with noise factors. The training dataset is composed of a preset number of samples selected in descending order of scores; the training optimization submodule is used to split the training dataset into multiple batches, filter high-value data based on the training loss of each batch of data and dynamically update the training batches to improve the convergence speed of large models.

[0011] As an improvement to the above system, the topic model is an LDA topic model, and the topic mining uses the Gibbs sampling algorithm; the original dataset Importance weight of Chinese text t for: ; in, Represents the original dataset The topic distribution of Chinese text t This represents the number of topics in the topic model. The text t represents the first... The probability of each topic; Represents the target dataset Thematic distribution; Importance score Calculation based on Gumbel sampling strategy: ; in, It is standard Gumbel noise.

[0012] As an improvement to the above system, the training optimization submodule is used to optimize the data within each batch. Data is obtained through forward propagation. training loss Based on training loss Calculated data Selection probability According to the selection probability Screening high-value data; among them, ; in, It is the number of losses. Indicates recent The cumulative distribution function composed of the loss of each training data point; It is a selectivity factor; When the selected high-value data reaches the preset batch size, backpropagation and parameter updates are performed on that batch of data.

[0013] To achieve another objective of the present invention, the present invention also provides a frequency track declaration coordination question-and-answer method based on a large model, implemented based on the aforementioned frequency track declaration coordination question-and-answer system based on a large model, comprising: Step 1: Obtain the raw data through the data processing module and perform preprocessing to build a pre-training dataset and a question-answering pair dataset; Step 2: Using the pre-training dataset as the original dataset and the question-answering pair dataset as the target dataset, the model training optimization module performs topic distribution modeling and text importance evaluation, and generates the training dataset by combining sampling and filtering strategies; the model training optimization module optimizes the training process of the large model based on data filtering and dynamic batch update strategies based on training loss, thereby completing the pre-training of the large model; Step 3: The model fine-tuning and deployment module uses a question-and-answer pair dataset to fine-tune the pre-trained large model and deploys the fine-tuned large model to realize the frequency track declaration coordination question-and-answer function.

[0014] The advantages of this application are: 1. A frequency and track application coordination question and answer system was designed, which solved the problems of lack of standard question and answer datasets and single optimization direction in the training process of large models in the existing satellite communication field; 2. It makes full use of professional data in the field of satellite communications, which can reduce the training time of the model while selecting high-quality training data, thereby improving the efficiency and accuracy of frequency and track application coordination. Attached Figure Description

[0015] Figure 1 The flowchart illustrates the frequency track application coordination question-and-answer method based on a large model provided in this embodiment of the invention. Detailed Implementation

[0016] The technical solutions provided in this application are further illustrated below with reference to the embodiments.

[0017] 1. Question-answering pair dataset construction 1.1 Data Acquisition and Format Conversion The Radio Regulations series of documents (including the Radio Regulations (Articles), Radio Regulations (Appendices), Radio Regulations (Resolutions and Recommendations), and Radio Regulations (ITU-R Recommendations with Citations Consolidated)) were used as data sources. OCR (Optical Character Recognition) technology was employed to extract text and tables from these documents, providing foundational data support for the subsequent construction of a frequency track declaration coordination Q&A dataset. The processed text data was stored in TXT format, and the table data in HTML format.

[0018] 1.2 Data structuring For the two types of data extracted by OCR (text data and tabular data), and considering the document structure characteristics of the "Radio Regulations" series of documents, a differentiated structured processing strategy is adopted to ensure that the data granularity meets the training requirements of large models: Text data processing: Since the text in the Radio Regulations is all marked with numerical labels as the starting identifier, the identifier naturally has the semantic rationality of paragraph division. In this embodiment, the numerical labels are used as the basis for division. The extracted text data is divided into several independent paragraphs according to the numerical labels. Each numerical label corresponds to an independent paragraph, and each paragraph forms an independent data record. Table data processing: According to the types of tables in the Radio Regulations, they can be divided into two categories: "row content tables" and "column content tables"; Row content table: Each row in the row content table can be considered a complete information entity, and the semantic relationship between different rows is relatively weak. During processing, the header information of each row in the row content table is concatenated with the specific content of that row to form independent data records. Column content tables: The content of each column in a column content table is closely related, while the semantic correlation between the content of different columns is not significant. During processing, the header information of each column in the column content table is combined with the content of all items in that column to form independent data records, thus completing the processing of the table data.

[0019] 1.3 Data Cleaning and Optimization Text and tabular data are combined and cleaned through steps such as rule filtering, data deduplication, and text segmentation to obtain the pre-training dataset required for subsequent large-scale model training.

[0020] 1.4 Target Problem Design Based on the above, by analyzing the Radio Regulations and existing frequency track application coordination materials, and with the guidance of field experts, we have identified the types of high-frequency query questions during the application coordination process. This embodiment constructs three categories with a total of nine specific questions that need to be queried during the application coordination process, covering core needs such as business consultation, limit query, and footnote retrieval, as shown in Table 1: Table 1: Issues related to frequency track application coordination:

[0021] 1.5 Automated Question-Answer Pair Generation Based on these questions, we wrote automated scripts to obtain the corresponding answer information from the pre-training dataset, complete the construction of the application coordination question-answer pair dataset, and provide direct, efficient, and high-quality data for the subsequent fine-tuning training of large models.

[0022] 2. Large Model Training Optimization Algorithm Design Having obtained the pre-training dataset needed for training the large model and the question-answer pair dataset needed for fine-tuning in the previous section, we will now train and optimize the large model. The large model training optimization algorithm mainly consists of two modules: a data selection module and a training optimization module.

[0023] 2.1 Data Selection Module: Used to accurately select samples from the raw pre-training data that highly match the frequency track application coordination task. The specific process includes: Corpus construction: using the pre-trained dataset as the original dataset The application coordination question-and-answer pair dataset will be used as a high-quality target dataset. The original dataset and target dataset As input corpus for LDA (Latent Dirichlet Allocation) topic model; Topic distribution modeling: The Gibbs sampling algorithm is used to mine topics from the input corpus, resulting in the original dataset. and target dataset Thematic distribution; Text importance quantification: Calculate the cosine similarity between topic distributions as the original dataset. Importance weight of Chinese text t :

[0024] in, Represents the original dataset The topic distribution of Chinese text t can be... Consider it as a A dimensional vector (m is the number of topics in the LDA topic model), The text t represents the first... The probability of each topic; Represents the target dataset Thematic distribution The target dataset The average value of the topic distribution for each text.

[0025] Gumbel sampling and filtering: After obtaining the original dataset Importance weight of Chinese text t Then, importance scores are calculated based on the Gumbel sampling strategy. :

[0026] in, It is standard Gumbel noise.

[0027] After calculating the original dataset Importance score for each text t Then, the original dataset All texts t are scored by importance Sort the data in descending order and select the top-k samples to form the training dataset for the model. Complete the data selection module.

[0028] 2.2 Training Optimization Module: By dynamically selecting high-loss data to prioritize parameter updates, the model convergence speed is improved, and the overall training cycle is shortened. When training large models, frequently adding data with significant loss to the training process can reduce training time. This embodiment constructs a training optimization module for large models based on this idea.

[0029] Training process breakdown: A round of training for a large model is divided into multiple steps, each of which involves three stages: forward propagation, back propagation, and parameter update on a batch of data.

[0030] Loss-driven data selection: For data within a certain batch Data is obtained through forward propagation. training loss Then, based on training loss Calculate the probability of selection To decide whether to let the data Perform the parameter update phase. Select probability. The calculation method is as follows:

[0031] in: It is the number of losses. Indicates recent The cumulative distribution function composed of the loss of each training data point; It is a selection factor used to adjust the selectivity of the selective backpropagation algorithm. The larger the value, the greater the probability that the training data will be selected.

[0032] Dynamic batch update: continuously updates data within the current batch. Filter the data; when selected... When the number of data points reaches the preset batch size, a high-value data subset is formed. Subsequent backpropagation is then performed on this high-value data subset to obtain the gradient of the model parameters. Then, the parameter update phase is carried out to complete the training of this batch of data.

[0033] Given a pre-training dataset and question-answer pairs, a high-quality training dataset is obtained after passing through the data selection module. Then, the training optimization module reduces the model training time and completes the training optimization of the large model.

[0034] 3. Design of Frequency Track Application Coordination Q&A System In the previous chapter, the implementation example completed the training of the large model and injected professional knowledge related to the field of satellite communications into the large model. Next, the application coordination question and answer pair dataset will be used as the fine-tuning dataset to fine-tune the full parameters of the large model, so that the large model can be aligned with the downstream application coordination tasks.

[0035] After fine-tuning, the large model is now capable of answering questions about frequency track application coordination. The large model will then be deployed to the server and the relevant front-end pages will be configured to complete the design of the frequency track application coordination Q&A system.

[0036] Based on the above description, it can be seen that the frequency track declaration coordination question-and-answer method based on a large model provided in this application, such as... Figure 1 As shown, it includes: Step 1: Obtain the raw data through the data processing module and perform preprocessing to build a pre-training dataset and a question-answering pair dataset; Step 2: Using the pre-training dataset as the original dataset and the question-answering pair dataset as the target dataset, the model training optimization module performs topic distribution modeling and text importance evaluation, and generates the training dataset by combining sampling and filtering strategies; the model training optimization module optimizes the training process of the large model based on data filtering and dynamic batch update strategies based on training loss, thereby completing the pre-training of the large model; Step 3: The model fine-tuning and deployment module uses a question-and-answer pair dataset to fine-tune the pre-trained large model and deploys the fine-tuned large model to realize the frequency track declaration coordination question-and-answer function.

[0037] The validity of this application will be verified through experiments as follows: 3.1 Input and Output Inputs: Radio Rules (RR) and Large Language Model (LLM); Output: Frequency Track Declaration Coordination Question and Answer System (FODC); 3.2 System Implementation Process 1. Perform OCR recognition on the RR to obtain its text and table data, and then preprocess the data to obtain the original dataset. ; 2. Write automation scripts in Q&A related to obtaining frequency track application coordination ; 3. and The corpus was used as a basis for topic modeling in LDA to obtain... Thematic distribution of each text and Theme distribution ; 4. Calculate based on the obtained topic distribution The importance score of Chinese text t is used to select data for the training dataset based on the Gumbel sampling strategy. ; 5. Split into several batch sets for easier training. 6. Perform the following operations for each batch: Perform forward propagation to obtain loss ,in, For the first One batch, ; For each loss ( ),according to Calculate the probability of selection ,according to This determines whether the current data is selected to participate in subsequent parameter update phases. For the first The loss of data; When data is selected When the quantity reaches the preset batch size, The gradients of the model parameters are obtained by backpropagation. During the parameter update phase, the parameter update is completed. Training.

[0038] 7. As a fine-tuning dataset, the pre-trained LLM is fine-tuned. Fine-tuning refers to training the pre-trained LLM using the fine-tuning dataset, thereby updating the parameters of the large model. 8. Deploy the fine-tuned LLM to the server, build the front-end page and connect it with the back-end large model to complete the implementation of FODC.

[0039] After constructing question-answer pairs, 102,699 pre-training data points and 187,219 question-answer pairs were obtained, followed by pre-training of a large model. To verify the effectiveness of the training optimization algorithm proposed in this embodiment, PyTorch training framework without any optimization methods and the data-optimized DSIR algorithm (Data selection for language models via importance resampling) were selected as comparison algorithms. For the proposed method and DSIR, both of which involve data selection, text with fewer than 20 characters or more than 500 characters in the two datasets were first filtered out. Then, the pre-training data was used as the original dataset, and the question-answer pairs were used as the target dataset for data selection, resulting in 80,000 data points for each dataset. For PyTorch, which does not involve data selection, 80,000 data points were randomly selected from the pre-training data as the pre-training dataset. This method selects two large models, Qwen3 1.7B and DeepSeek R1 1.5B, as base models for experiments. The graphics cards used for training are two RTX 4090s. The specific training parameters are obtained based on the characteristics of the dataset, the choice of model, previous experience, and multiple parameter tuning comparisons during the experiment. The parameter list is shown in 2. Table 2: Training parameters of the model

[0040] After model pre-training, this method uses question-answer pairs to fine-tune the model to evaluate the performance of the large model. The question-answer pairs are split into training and validation sets in a 9:1 ratio. The fine-tuning process consists of two rounds with a batch size of 16, and the default AdamW optimizer is used. This method is compared with two comparative algorithms based on BLEU score, ROUGE1 score, and training time. BLEU and ROUGE1 scores are commonly used evaluation metrics for question-answering systems, and training time refers to the time spent training the model on the pre-training dataset. Table 3 shows the results of each algorithm on different pre-trained models: Table 3: Comparison of experimental results for each algorithm

[0041] As shown in Table 3, the proposed method outperforms other comparative algorithms overall. Compared with PyTorch and the data-driven optimization algorithm DSIR, this method models the text at the topic level while optimizing training time using selective backpropagation. Therefore, the algorithm can significantly improve training time while maintaining high BLEU and ROUGE1 scores: on the Qwen model, the BLEU score decreases by an average of only 5.5%, and the F1 score decreases by an average of only 6.8%, but the training time is reduced by an average of 16.2%. This phenomenon is even more pronounced on the more complex DeepSeek model: the BLEU score decreases by an average of 2.9%, the ROUGE1 score decreases by an average of 3.1%, and the training time is reduced by an average of 16.5%. The experimental results demonstrate the effectiveness of the proposed method in terms of model performance and training time.

[0042] The training and fine-tuning of the large model have now been completed. Next, the large model will be deployed to a Linux server as the system backend, and Streamlit will be used as the system frontend to complete the design of the frequency track declaration coordination Q&A system.

[0043] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A frequency track declaration coordination question-and-answer system based on a large model, characterized in that, include: The data processing module is used to acquire raw data and preprocess it to build a pre-training dataset and a question-answering pair dataset. The model training optimization module is used to take the pre-training dataset as the original dataset and the question-answer pair dataset as the target dataset, and generate the training dataset by combining topic distribution modeling and text importance evaluation with sampling and filtering strategies. It is also used to optimize the training process of large models based on data filtering and dynamic batch update strategies based on training loss, thereby completing the pre-training of large models; and The model fine-tuning and deployment module is used to fine-tune the pre-trained large model using a question-and-answer pair dataset, and then deploy the fine-tuned large model to realize the frequency track declaration coordination question-and-answer function.

2. The frequency track declaration coordination question-and-answer system based on a large model according to claim 1, characterized in that, The raw data includes: a series of radio regulations documents.

3. The frequency track declaration coordination question-and-answer system based on a large model according to claim 1, characterized in that, The data processing module includes a data acquisition and format conversion unit, a data structuring processing unit, a data cleaning and optimization unit, and a question-answer pair generation unit; wherein... The data acquisition and format conversion unit extracts and stores the text and tabular data from the original data. The data structuring processing unit is used to convert the text data and tabular data of the original data into independent data respectively; The data cleaning and optimization unit is used to clean the transformed data through rule filtering, data deduplication, and text segmentation steps to obtain the pre-training dataset. The question-answer pair generation unit generates the question-answer pair dataset by extracting corresponding answers from the pre-trained dataset through an automated script based on the frequency track declaration and coordination of high-frequency query question types.

4. The frequency track declaration coordination question-and-answer system based on a large model according to claim 3, characterized in that, The data structuring processing unit is used to divide text data based on paragraphs, converting each paragraph into independent data; it is used to concatenate the table header information of the row content of the table data with the specific content of the row to which the header information belongs, thereby converting the row content table into independent data; it is used to concatenate the table header information of the column content of the table data with the content of the column to which the header information belongs, thereby converting the column content table into independent data.

5. The frequency track declaration coordination question-and-answer system based on a large model according to claim 3, characterized in that, The frequency track declaration coordination high-frequency query question types include: business consultation, limit value query, and footnote retrieval.

6. The frequency track declaration coordination question-and-answer system based on a large model according to claim 1, characterized in that, The model training optimization module includes: a data selection submodule and a training optimization submodule; among which, The data selection submodule is used to select the pre-trained dataset as the original dataset. The question-answer pair dataset is used as the target dataset. By modeling topic distribution and evaluating text importance, a topic model is used to analyze the original dataset. With the target dataset Topic mining was performed to obtain the original datasets. and target dataset Topic distribution; used to calculate the original dataset The similarity between the topic distribution of the Chinese text t and the target dataset is used as the basis for the original dataset. Importance weight of Chinese text t Used to calculate importance scores in conjunction with noise factors. The training dataset is composed of a predetermined number of samples selected in descending order of scores. The training optimization submodule is used to split the training dataset into multiple batches, filter high-value data based on the training loss of each batch of data, and dynamically update the training batches to improve the convergence speed of large models.

7. The frequency track declaration coordination question-and-answer system based on a large model according to claim 6, characterized in that, The topic model is an LDA topic model, and the topic mining uses the Gibbs sampling algorithm; The original dataset Importance weight of Chinese text t for: ； in, Represents the original dataset The topic distribution of Chinese text t This represents the number of topics in the topic model. The text t represents the first... The probability of each topic; Represents the target dataset Thematic distribution; Importance score Calculation based on Gumbel sampling strategy: ； in, It is standard Gumbel noise.

8. The frequency track declaration coordination question-and-answer system based on a large model according to claim 6, characterized in that, The training optimization submodule is used to optimize the data within each batch. Data is obtained through forward propagation. training loss Based on training loss Calculated data Selection probability According to the selection probability Screening high-value data; among them, ； in, It is the number of losses. Indicates recent The cumulative distribution function composed of the loss of each training data point; It is a selection factor; When the selected high-value data reaches the preset batch size, backpropagation and parameter updates are performed on that batch of data.

9. A frequency track declaration coordination question-and-answer method based on a large model, implemented based on the frequency track declaration coordination question-and-answer system based on any one of claims 1-8, comprising: Step 1: Obtain the raw data through the data processing module and perform preprocessing to build a pre-training dataset and a question-answering pair dataset; Step 2: Use the pre-training dataset as the original dataset and the question-answer pair dataset as the target dataset. Use the model training optimization module to model topic distribution and evaluate text importance, and combine it with sampling and filtering strategies to generate the training dataset. The model training optimization module optimizes the training process of the large model based on data filtering and dynamic batch update strategies using training loss, thereby completing the pre-training of the large model. Step 3: The model fine-tuning and deployment module uses a question-and-answer pair dataset to fine-tune the pre-trained large model and deploys the fine-tuned large model to realize the frequency track declaration coordination question-and-answer function.

Images