Digital intelligent treatment auxiliary method based on generative AI large model

Abstract

The invention discloses a digital intelligent management auxiliary method based on a generative AI large model, and solves the problems of high cost of long Prompt reasoning, reduction of output quality and the like by selecting high-quality fine tuning data and designing an effective fine tuning strategy. According to the technical scheme, from pre-training, continuous pre-training and field alignment, model illusion is reduced by recalling knowledge, the answer timeliness is ensured, and model answering is quickly intervened, and high-quality large-scale field fine tuning data is constructed by adopting data generation methods such as Self-Instruct, Self-QA and Self-KG, so that the performance of the model in a specific field is improved. Besides, Prompt tokens are added to each layer of the large model by using P-Tuning-V2 to improve the number of trainable parameters and performance of the model, and the autonomous large model is constructed and fine-tuned to ensure data security and meet service requirements, and meanwhile, the dependence on external models is reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and particularly relates to a digital governance assistance method based on a generative AI large model. Background Art

[0002] With the advent of the big data era, data is growing at an astonishing speed, providing an unprecedented information treasure trove for enterprises and social organizations. The importance of data governance for this data deluge has become increasingly prominent, making it a key factor in the development of enterprises and society. Traditional data governance methods often rely on manual review and rule-making, which are inefficient and error-prone, affecting the accuracy and effectiveness of data governance. In recent years, the rapid development of generative AI (Artificial Intelligence) technology has provided new solutions for data governance. Generative AI large models can automatically learn the inherent laws of data from massive amounts of data and generate high-quality data samples, providing a new perspective and strong support for data governance. This technology can not only significantly improve the efficiency and accuracy of data governance but also dynamically adjust and optimize according to actual needs, enhancing the flexibility and adaptability of data governance.

[0003] Most of the current data governance tools on the market still rely on process control technology, database technology, and traditional machine learning technology, and they have many limitations in processing data. With the rapid development of generative AI technology, large language models (LLMs) such as the GPT (Generative Pre-trained Transformer) series have demonstrated powerful capabilities in the field of natural language processing (NLP). These large models have mastered rich language knowledge and common sense through massive amounts of pre-training data and can provide high-quality text generation and understanding services for various tasks. However, directly applying general large models to vertical fields such as digital governance often faces problems such as poor domain adaptability and unstable output quality.

[0004] To solve these problems, the industry generally adopts two methods: Prompt Engineering and Fine-tuning. Prompt Engineering guides the model to produce desired outputs by designing clever prompts, but it has high requirements for the length and accuracy of the prompts and is prone to exceeding the processing capacity of the large model. In contrast, Fine-tuning can more stably improve the model performance by adjusting the model parameters to adapt to specific domain data. However, traditional fine-tuning methods often require a large amount of labeled data, and the adjustment process is complex and difficult to adapt to rapidly changing business needs.

[0005] In summary, the current existing technologies mainly have the following disadvantages and problems: 1. Limitations of Prompt Engineering: Although Prompt Engineering is a relatively easy - to - get - started way of using large models, it highly depends on manually designed Prompts. This dependence not only requires designers to have rich experience and professional knowledge to ensure that the Prompt can accurately guide the model to produce the expected output, but also faces the challenge of being difficult to maintain effectiveness in all scenarios. In addition, due to the limited length of the input sequence for large models, overly long Prompts will significantly increase the inference cost of the model, resulting in a decrease in output quality; in extreme cases, overly long Prompts may even be truncated, further affecting the output accuracy of the model.

[0006] 2. High data demand and quality requirements: Whether it is Prompt Engineering or traditional fine - tuning methods, they all have high requirements for data. High - quality fine - tuning training data is the key to ensuring the stable performance of the model in specific tasks or domains. However, collecting and annotating a sufficient amount of high - quality data is often a daunting task, which is not only time - consuming and labor - intensive, but also may face sensitive issues such as data privacy and security; in the case of limited data, the improvement brought by fine - tuning may be negligible and unable to meet the needs of practical applications.

[0007] 3. Complex fine - tuning process: When fine - tuning a generative AI large model for domain - specific applications, the operation complexity increases significantly. The fine - tuning process not only requires rich data science knowledge and skills, but also in - depth understanding of the model architecture and training mechanism. Especially when dealing with multi - modal data or complex problems in specific domains, selecting appropriate fine - tuning strategies, adjusting parameters, and monitoring the training process all require a high level of professionalism. In addition, due to the large scale of the model, the demand for computing resources is also very high, which further increases the difficulty and cost of fine - tuning; therefore, for most enterprises and organizations, performing efficient fine - tuning operations is a huge challenge.

[0008] 4. Poor adaptability of general large models in specific domains: General large models often show significant limitations in adaptability when facing specific domains. First of all, it is difficult for them to accurately understand and respond to professional terms in the domain, and these terms may not be fully trained or lack corresponding context understanding in general models; secondly, for complex problems, especially those that require combining domain knowledge and reasoning ability, the answers of general large models often seem inadequate and difficult to capture the deep meaning and domain - specific details of the problem; this poor adaptability problem not only affects the accuracy and reliability of the model in specific domains, but also limits its wide application and in - depth development in professional fields.

[0009] Therefore, in the context of the big data era, how to make full use of the advantages of generative AI big models and develop a data governance technology that can comprehensively, efficiently and flexibly process various types of data has become an important issue that needs to be urgently addressed in the current data governance field. Summary of the invention

[0010] In view of the above, the present invention provides a digital governance assistance method based on a generative AI big model to overcome the bottleneck problem of the generative AI big model in the application of vertical fields such as digital governance.

[0011] A digital intelligence governance auxiliary method based on a generative AI big model includes the following steps: (1) Collect massive multi-source heterogeneous data from multiple channels, pre-process the data and divide it into training set, validation set and test set; (2) Select a large language model based on generative AI and optimize it according to the specific needs of digital governance scenarios through fine-tuning technology; (3) Adjusting the large language model structure according to task requirements and generating vertical field fine-tuning data; (4) Use the Prompt Engineering technology to train the large language model using the training set data, and use the P-Tuning (Parameter Tuning) technology to fine-tune the model during the training process using the vertical field fine-tuning data; (5) Input the test set data or real-time data into the trained model for predictive analysis, and provide users with governance support decisions based on the predictive analysis results output by the model.

[0012] Furthermore, the massive multi-source heterogeneous data collected in step (1) includes public data (such as economic indicators, policy documents), internal enterprise data (such as business data, financial statements), social media data (such as user comments), and real-time data generated by IoT devices.

[0013] Furthermore, the data preprocessing in step (1) includes four parts: data cleaning, data standardization, data enhancement and feature engineering. The data cleaning part includes denoising, filling in missing values, and correcting or eliminating outliers; data standardization includes data type conversion, data encoding, data aggregation, and normalization processing; the feature engineering part includes feature selection and feature extraction, wherein feature selection uses statistical methods to screen out key features that have an important impact on governance decisions, and feature extraction uses machine learning methods to automatically extract high-level features from raw data.

[0014] Preferably, the large language model in step (2) adopts the GPT series models based on Transformer (generator), which have been pre-trained on large-scale datasets; in order to inject professional knowledge in the field of digital governance, the fine-tuning technology continues the pre-training on top of this model. By inputting mixed data, namely domain data and a small amount of general data, it ensures that while maintaining the general capabilities, the model enhances its understanding and application capabilities of domain knowledge; then, through SFT (Supervised Fine-Tuning) and RLHF (Reinforcement Learning from Human Feedback), the model's understanding and answering capabilities for domain problems are further improved, making the model output more in line with people's preferences and expectations. The GPT series models have the superiority of the Transformer architecture, the ability of large-scale pre-training, and the potential of zero-shot and few-shot learning. In terms of policy document generation and optimization, the GPT model can assist policymakers in generating normative texts such as policy drafts and legal provisions. By inputting relevant policy objectives, background information, and constraints, the model can automatically generate preliminary policy texts and provide multiple expression methods. In the field of government services, the GPT model can build an efficient intelligent Q&A system to provide the public with consulting services on policies, regulations, etc. Users can ask questions in natural language, and the system uses the powerful understanding ability of the GPT model to quickly retrieve relevant information and give accurate answers. By analyzing public opinions and sentiment trends on channels such as social media and forums, the model can assist decision-makers in understanding social dynamics and hot issues. At the same time, the model can also use historical data and current trends for predictive analysis to provide a scientific basis for policy formulation and adjustment. In a broader governance field, the GPT model can also be part of an auxiliary decision-making support system. By integrating and analyzing data and information from different channels, the model can provide comprehensive decision-making references and suggestions for decision-makers. For example, in the fields of urban planning, environmental protection, and public safety, the GPT model can generate multiple possible solutions and prediction results to help decision-makers make more scientific and reasonable decisions.

[0015] Furthermore, the specific method for adjusting the large language model structure in step (3) is as follows: For classification tasks, one or more fully connected layers need to be added on top of the pre-trained model. These layers map the hidden representations of the model to the class space required by the task and use the activation function Softmax to output the prediction probabilities for each class; For regression tasks, a single fully connected layer is used as the output layer of the model, and the activation function is selected as ReLU, Sigmoid, or linear activation according to the range of the output values to directly output continuous values; For sequence labeling tasks, a CRF (Conditional Random Field) layer needs to be added to the model, which considers the dependencies between labels to improve the accuracy of labeling; Finally, according to the specific requirements of the task, the input layer and output layer of the model need to be adjusted accordingly. For variable-length input data, additional processing mechanisms including padding and bucketing are introduced to ensure that the model can handle data of different sizes.

[0016] Furthermore, the specific method for generating vertical domain fine-tuning data in step (3) is as follows: when there are some seed instruction data, the Self-Instruct + Qwen (Qwen is a high-performance large language model developed by Alibaba) method is used for expansion to generate more fine-tuning data that relatively meets the requirements. Each piece of fine-tuning data includes three parts: instruction, input, and output; when there is no basic seed instruction data, the Self-QA + Qwen method is used to directly generate instruction data from the document. Qwen analyzes and understands the unstructured document to generate instructions and answers related to the document content, thereby constructing an instruction dataset for fine-tuning; when there is a knowledge graph, the Self-KG + Qwen method is used to directly generate instruction data based on the knowledge graph. This method makes full use of the structured information in the knowledge graph to generate instructions and answers closely related to the digital governance field, providing strong data support for model fine-tuning.

[0017] Furthermore, the specific method for training the large language model using Prompt Engineering technology in step (4) is as follows: first, design clear, concise, and targeted Prompts to stimulate the model to generate appropriate outputs. The Prompts should include clear task instructions, relevant context, example references, and the expected output format; then, use the validation set and feedback loop method to continuously optimize the effects and performance of the Prompts, try different Prompt combinations and variants, and select the best Prompts by comparing their performances; finally, based on the training set data and Prompts, combine zero-shot learning, few-shot learning, chain-of-thought prompting, and transfer learning methods to train the large language model, and use the Prompts to understand and complete the training tasks.

[0018] The role of Prompt Engineering technology in model training mainly includes three points: First, it improves model performance. Through carefully designed prompts, the model can be guided to better learn and understand tasks during training, thereby enhancing the performance and effectiveness of the model. It can help the model improve its generalization ability and accuracy through context learning and other methods with limited data. Second, it reduces training time and costs. Prompt Engineering does not require modifying the weights or parameters of the model. Only the prompts need to be adjusted externally, so it can significantly reduce training time and computational resource consumption. Third, it enhances the interpretability and controllability of the model. By optimizing the prompts, the output of the model can be made more in line with expectations, while improving the interpretability of the model, making it easier for users to understand the decision-making process of the model. It can also help users better control the output of the model to meet specific requirements or limitations.

[0019] Furthermore, in step (4), the P-Tuning-V2 (P-Tuning version 2) technology is adopted to add Prompts tokens (prompt tokens) together with data as input before each layer of the large language model, rather than only adding them in the Embedding layer. The Prompts tokens are a fixed-length trainable vector, and only the parameters of this vector are updated during fine-tuning. This not only increases the number of learnable parameters but also makes the Prompts connected to deeper layers, having a more direct impact on the model output, thereby improving the adaptability and accuracy of the model in the field of digital governance.

[0020] Furthermore, an intelligent feedback mechanism is incorporated into the model training in step (4). This mechanism collects user feedback information and system operation data comprehensively, analyzes key performance indicators and abnormal information, and precisely optimizes the model and algorithm, including adjusting model parameters, reconstructing algorithm logic, or introducing cutting-edge technologies, to improve the efficiency and accuracy of the model. This enables the model to keep up with market trends and technological development, implementing a system iteration upgrade strategy aimed at continuously integrating new functions, optimizing the user interaction experience, and enhancing the overall system performance, bringing a more excellent user experience to users.

[0021] Furthermore, in step (5), the test set data or real-time data is input into the trained model for predictive analysis. The model automatically learns the inherent laws of the data with its powerful data processing capabilities, complex pattern recognition and creative content generation capabilities, and mines out hidden correlations, trends and potential risk points, generates customized, high-precision decision suggestions and predictive analysis results, and presents them to users in the form of interactive charts, dynamic simulation scenarios or directly generated decision reports, providing users with governance support decisions; at the same time, domain expert knowledge is integrated, that is, by writing rule sets, defining logic and constructing domain ontology, combined with model prediction results, and then improving the accuracy and reliability of governance support decisions through logical reasoning and verification. For example, in certain scenarios of digital governance, expert knowledge bases can be used to assist in judgment, prediction or recommendation to ensure the rationality and professionalism of system output. This intelligent decision-making assistance method not only greatly improves the efficiency and accuracy of the decision-making process, but also effectively reduces the subjectivity and uncertainty of human judgment, providing a strong scientific basis and forward-looking guidance for governance decisions of enterprises and all sectors of society.

[0022] Based on the above technical solution, the present invention can solve the following technical problems: 1. Solve the limitations of Prompt Engineering; Prompt Engineering is easy to use but has problems such as high inference cost, limited output quality, unstable effect, and difficulty in handling personalized service needs. This invention introduces a strategy based on high-quality domain fine-tuning data. By building a large-scale, high-quality domain-specific fine-tuning dataset, it reduces the dependence on long prompts, reduces the inference cost, and improves the accuracy and stability of the output. In addition, in response to personalized service needs, this invention supports the use of own data to train lightweight fine-tuning models for each user, ensuring the flexibility and adaptability of the model.

[0023] 2. Respond to large-scale data needs; Generating high-quality domain fine-tuning data requires large-scale, multimodal data sets, and manually collecting and compiling these data is time-consuming and expensive. This paper adopts a variety of data generation methods such as Self-Instruct, Self-QA and Self-KG, which can efficiently generate fine-tuning data that meets the requirements from existing data in an automated or semi-automated manner, significantly reducing the cost of data collection; at the same time, these methods can generate rich single-round / multi-round instruction fine-tuning data, providing sufficient data support for model training.

[0024] 3. Simplify the adjustment process and improve efficiency; The traditional fine-tuning method is complex to operate, requires rich professional knowledge and experience, and the adjustment effect is greatly affected by the quality and quantity of the dataset. The present invention adopts a fine-tuning method based on P-Tune (parameter fine-tuning), and realizes in-depth adjustment of model parameters by adding new parameters before the Embedding layer and each layer of the large model; This method not only simplifies the adjustment process and improves efficiency, but also enhances the model's adaptability to data in specific fields by increasing the number of trainable parameters, thereby improving the fine-tuning effect.

[0025] 4. Enhance domain adaptability; General large models often have poor adaptability in specific domain applications and are difficult to accurately understand and respond to professional terms and complex problems within the domain. The present invention injects rich domain knowledge into the model through a combination of continued pre-training and fine-tuning; In the continued pre-training stage, the general ability of the model is maintained by mixing data; In the fine-tuning stage, high-quality domain fine-tuning data is used to deeply customize the model. In addition, the present invention also considers subtle differences such as the writing style and tone of the model, and makes the model show more professional and accurate behavior in specific domains through fine-tuning. These measures together enhance the adaptability and performance of the model in vertical domains such as digital governance.

[0026] Therefore, the innovation and beneficial technical effects of the present invention are mainly reflected in the following aspects: 1. Efficient LLM fine-tuning.

[0027] In the prior art, although the Prompt Engineering method is easy to get started, it is limited by the high inference cost and unstable output quality caused by the Prompt length. The present invention realizes fine-grained fine-tuning of LLM parameters by introducing the P-Tuning-V2 method, which not only overcomes the limitation of the Prompt length, but also significantly improves the stability and output quality of the model. Compared with the traditional Prompt Engineering, P-Tuning-V2 greatly increases the number of learnable parameters by adding Prompt tokens to all layers, enabling the model to better adapt to the subtle differences and complex patterns in specific domains.

[0028] 2. Generation of high-quality domain fine-tuning data.

[0029] Aiming at the problem of difficult construction of domain fine-tuning data, the present invention innovatively proposes three data generation methods: Self-Instruct, Self-QA, and Self-KG. These methods can automatically generate a large amount of qualified fine-tuning data based on a small amount of existing data or knowledge graphs, effectively solving the problem of data collection, and significantly improving the quality and scale of fine-tuning data.

[0030] 3. Precise adaptation of large models in vertical domains.

[0031] Different from the practice of directly applying general large models in the prior art, the present invention conducts continued pre-training and fine-tuning on the basis of large language models. By inputting mixed data and domain-specific corpora, the model not only maintains general capabilities but also acquires domain knowledge. In addition, through SFT and RLHF technologies, the present invention further improves the model's ability to understand problems and answer questions within the domain and its alignment ability with people's preferences, thereby ensuring the precise adaptation of the model in the field of digital and intelligent governance.

[0032] 4. Enhancement of data security and privacy protection.

[0033] Regarding the data security issue, the present invention proposes a solution of building its own large model and conducting fine-tuning. This solution not only ensures that data will not be transmitted to third-party large model services but also improves the model's capabilities in specific domains through fine-tuning technology, meeting business requirements while strengthening data security and privacy protection.

[0034] 5. Enhanced model customization and flexibility.

[0035] The present invention fine-tunes the LLM to adapt to specific tasks or domains, enabling the model's behavior to better adapt to specific nuances, tones, or terms. This customization not only improves the model's accuracy and relevance but also enhances the model's flexibility, enabling it to be quickly adjusted and optimized according to the needs of different users. Brief Description of the Drawings

[0036] Figure 1 It is a schematic diagram of the system implementation framework of the digital and intelligent governance assistance method based on the generative AI large model of the present invention.

[0037] Figure 2 It is a schematic diagram of the implementation route of the large model in the vertical domain of the present invention. Detailed Embodiment

[0038] To describe the present invention more specifically, the technical solutions of the present invention will be described in detail below in conjunction with the drawings and specific embodiments.

[0039] Select a model that has been pre-trained on a large-scale dataset. The model used in this embodiment is the GPT series model. Such a model has learned useful features and patterns on a wide range of data and tasks. At the same time, prepare a dataset related to the target task in the governance scenario. These data are labeled training data, validation data, and test data, and the preprocessing operation of the data is completed at the same time. After the dataset is prepared and processed perfectly, the digital and intelligent governance assistance system based on the generative AI large model is implemented through the following process, asFigure 1 As shown below: (1) Data collection and processing.

[0040] The large model data layer is the cornerstone of the entire digital intelligence governance assistance system, responsible for collecting and integrating data from multiple channels. This data is the basis for subsequent processing and analysis and is crucial for the accuracy and effectiveness of the system.

[0041] The system needs to be able to access various types of data sources, which are diverse in origin, including but not limited to public data from social institutions (such as economic indicators, policy documents), enterprise internal data (such as business data, financial statements), social media data (such as user comments), and real-time data generated by Internet of Things devices, etc. In addition, the system stipulates a unified data interface standard and strictly controls data quality. To ensure that data from different sources can smoothly enter the system and be effectively utilized, it is necessary to formulate a unified data interface standard and data exchange protocol, which includes standardization in aspects such as data format, transmission rate, and security authentication. During the data collection process, strict quality control measures must be implemented to ensure the accuracy, integrity, and consistency of the data, which may include steps such as data verification, deduplication, and outlier handling.

[0042] The data processing layer is responsible for preprocessing and feature extraction of the collected raw data to provide high-quality data input for subsequent machine learning model training, including preprocessing steps such as data cleaning, formatting, and normalization, as well as data augmentation techniques for specific tasks, ensuring the accuracy, consistency, and effectiveness of the data.

[0043] 1.1 Data cleaning: This step aims to remove noise, errors, and redundant information in the data, which may include operations such as filling missing values, deleting invalid records, and correcting incorrect data. When dealing with missing values, the mean, median, mode filling, or model-based prediction is used to fill the missing values to reduce the incompleteness of the data; in terms of outlier detection and correction, statistical methods such as Z-score (standard score) or model-based methods are used to identify outliers and decide whether to delete, correct, or retain these outliers; when encountering data duplicates, hash tables, sorting and comparison methods, etc. are used to identify and delete duplicate records to maintain the uniqueness of the data.

[0044] 1.2 Data Formatting: Convert data from different sources into a unified format and structure for subsequent processing and analysis. This part mainly includes operations such as data type conversion, data encoding, and data aggregation. When performing data type conversion, according to the requirements of the model, use the bag-of-words model or Word2Vec (word vector model) to convert text data into numerical type, or convert date and time data into timestamps or specific formats; in terms of data encoding, encode categorical variables, use one-hot encoding to process categorical data, or convert ordinal categorical variables into integer encoding; in terms of data aggregation, according to business requirements, perform grouping and aggregation operations on data, such as calculating the average value, maximum value, etc. according to time windows.

[0045] 1.3 Data Normalization and Standardization: In order to eliminate the dimensional differences and scale effects between different features, it is necessary to normalize the data. This helps to speed up the model training speed and improve the model performance. Adopt the min-max normalization method to scale the features between the given minimum and maximum values (0 and 1), which is suitable for cases where the distribution is unknown; in addition, adopt the Z-score standardization method to convert the features into a distribution with a mean of 0 and a standard deviation of 1 by subtracting the mean and dividing by the standard deviation, which is suitable for data that conforms to the Gaussian distribution.

[0046] 1.4 Feature Extraction: Extract key features useful for governance decisions from the preprocessed data. This requires applying domain knowledge, statistical methods, and machine learning algorithms to identify and select the most representative features for subsequent model training. In this invention, mainly extract features such as images, texts, and time series; in addition, still need to select the features that have the greatest impact on the model performance through filtering methods, wrapper methods, or embedding methods.

[0047] (2) Model Selection and Training.

[0048] The model layer is the core part of the digital intelligence governance assistance system, responsible for training and optimizing using generative AI large models. It deeply integrates and efficiently utilizes advanced generative AI large models, which are built based on massive data and complex algorithms and have powerful learning and creation capabilities. At this level, the system continuously iteratively optimizes the model parameters through carefully designed training strategies and continuous optimization algorithms to improve its accuracy, generalization ability, and the ability to handle complex problems. At the same time, this layer also incorporates an intelligent feedback mechanism to automatically adjust the training plan according to the training effect to ensure that the model can accurately adapt to the changing actual application scenario requirements.

[0049] 2.1 Selection and Fine-tuning of Large Language Models: First, in this embodiment, a large language model based on generative AI is selected as the base model. Given the advantages of large language models in handling complex tasks and generating high-quality content, we optimize the model for the specific needs of the digital governance field through fine-tuning techniques. The necessity of fine-tuning lies in its ability to effectively improve the stability and output quality of the model on specific datasets and tasks, avoiding the problems of increased inference costs and decreased output quality caused by overly long Prompts.

[0050] Select a suitable generative AI large model according to the characteristics and requirements of the governance field, which includes models based on Transformer (such as GPT series models), generative adversarial networks (GANs), or other advanced generative models. Transformer is a neural network architecture based on the self-attention mechanism. GPT series models are the specific implementations of the Transformer architecture in generative tasks. These models learn rich language knowledge and common sense through unsupervised pre-training on a vast amount of text data. During the pre-training process, the goal of the model is to predict the next word or sentence based on the previous text. This training method enables GPT series models to possess powerful language generation capabilities.

[0051] As an outstanding representative of the Transformer architecture, GPT series models have revolutionized the field of natural language processing through their unique encoder structure. The self-attention mechanism is the core of this architecture, which allows the model to consider all other words in the sequence when processing each word in the sequence, thus effectively capturing long-range dependencies. This ability is crucial for understanding complex language structures and contexts, enabling GPT series models to excel in generating coherent and logically rigorous texts.

[0052] During the training process of GPT series models, a vast amount of text data on the Internet is utilized. These data cover a wide range of topics and contexts. Through pre-training on large-scale datasets, the model has not only learned basic language rules but also absorbed rich language knowledge and common sense, including multiple meanings of vocabulary, grammatical structures, idiomatic expressions, etc. The result of this deep learning enables GPT models to generate texts that are both in line with language norms and creative. Another remarkable feature of GPT series models is their powerful zero-shot and few-shot learning capabilities, which means that even in the absence or with only a small amount of labeled data, the model can give reasonable responses to new tasks based on existing knowledge and understanding. This ability is particularly important for the governance field because tasks such as policy-making and legal consulting often involve a large number of unseen new situations and problems, and GPT models can quickly adapt and give valuable suggestions or answers.

[0053] A generative adversarial network (GAN) is a deep learning model that learns through the mutual game between two neural networks, namely the generator and the discriminator. This unsupervised learning method has achieved remarkable results in multiple fields. The generator is a neural network whose input is a latent vector, usually following a certain random distribution (such as Gaussian distribution or uniform distribution); the goal of the generator is to learn the distribution of real data and generate fake data similar to the real data. It continuously adjusts its parameters so that the generated fake data becomes increasingly difficult to be distinguished by the discriminator. The discriminator is also a neural network whose input is real data or fake data generated by the generator. The goal of the discriminator is to distinguish whether the input data is real or fake data generated by the generator. It optimizes itself by maximizing the classification accuracy, that is, to judge the authenticity of the input data as accurately as possible.

[0054] The learning process of GAN is a zero-sum game process, in which the generator and the discriminator confront each other and continuously adjust their parameters; during the training process, the generator tries to generate increasingly real fake data to deceive the discriminator, while the discriminator continuously improves its discrimination ability to distinguish real data and fake data. This adversarial process prompts the two networks to continuously improve their abilities and finally reach an equilibrium point, that is, the discriminator cannot accurately judge whether the fake data generated by the generator is true.

[0055] 2.2 Continue pre-training and domain alignment: As Figure 2 shown, in order to inject professional knowledge in the field of digital intelligence governance, this implementation method conducts continue pre-training on the basis of the basic large model. By inputting mixed data (domain data and a small amount of general data), it ensures that the model enhances its understanding and application ability of domain knowledge while maintaining general capabilities. In addition, through techniques such as SFT and RLHF, we further improve the model's understanding and answering ability for domain problems, making the model output more in line with people's preferences and expectations.

[0056] 2.3 Adjust the model structure: According to the requirements of the task, it may be necessary to make some modifications to the structure of the pre-trained model. For example, it may be necessary to add a classification layer or regression layer for specific tasks or adjust model parameters, specifically: (1) Analyze task characteristics: First, it is necessary to clearly define the nature of the task, whether it is a classification task, a regression task, or a more complex sequence labeling, generation task, etc.; different tasks have completely different requirements for the model output. The classification task requires the model to output the probability distribution belonging to a certain category, while the regression task requires the model to output continuous values. In addition, it is necessary to consider the nature of the data, such as text length, image size, etc., which will all affect the design of the model input layer.

[0057] (2) Add or modify specific layers: Classification layer: For classification tasks, usually one or more fully connected layers need to be added on top of the pre-trained model. These layers map the hidden representations of the model to the class space required by the task; the activation function usually selects Softmax to output the prediction probabilities for each class.

[0058] Regression layer: For regression tasks, a single fully connected layer may be used as the output layer, and the activation function is selected according to the range of output values (such as ReLU, Sigmoid or linear activation) to directly output continuous values.

[0059] Sequence labeling layer: For sequence labeling tasks such as named entity recognition (NER), a specific type of layer such as a CRF layer may need to be added, which takes into account the dependencies between labels to improve the accuracy of labeling.

[0060] Adjust the input / output layers: According to the specific requirements of the task, it may also be necessary to make corresponding adjustments to the input and output layers of the model. For example, for variable-length input data (such as texts or images of different lengths), additional processing mechanisms such as padding and bucketing may need to be introduced to ensure that the model can handle data of different sizes.

[0061] (3) Adjust model parameters: After adding or modifying layers, it may also be necessary to adjust the hyperparameters of the model, such as the learning rate, batch size, number of training epochs, etc., to optimize the training process and model performance. These adjustments usually need to be gradually optimized based on experiments and experience.

[0062] 2.4 Construction of large models in vertical domains: In the implementation of vertical domains (digital governance), this implementation follows the phased process of pre-training, continued pre-training, and alignment, and gradually constructs a large model suitable for the digital governance domain. Through continuous iteration and optimization, it ensures that the model has high accuracy and professionalism in the digital governance domain.

[0063] 2.5 Model training based on Prompt Engineering: In the model training layer module, this implementation uses large model Prompt Engineering. Prompt Engineering is an important technology used to optimize and manage the prompts used to train or guide AI models to ensure that the model can accurately and efficiently execute user instructions or tasks; the applications of Prompt Engineering in the model training layer include: Clear Prompt Design: During model training, it is necessary to design clear, concise, and targeted Prompts to stimulate the model to generate appropriate outputs; the Prompt should include elements such as clear task instructions, relevant context, example references, and the expected output format.

[0064] Prompt Optimization: Utilize techniques such as validation sets and feedback loops to continuously optimize the effectiveness and performance of the Prompt. Different Prompt combinations and variants can be tried, and the best solution can be selected by comparing their performance.

[0065] Application of Prompt Techniques: At the model training layer, various Prompt techniques such as zero-shot learning, few-shot learning, and chain-of-thought prompting can be introduced. These techniques can help the model understand and complete tasks through prompts without a large amount of training data.

[0066] Prompt Management: As model training progresses and tasks become more complex, a large number of Prompts need to be managed. This includes organizing, storing, and retrieving Prompts so that they can be quickly found and used when needed.

[0067] (3) Generation of vertical domain fine-tuning data.

[0068] The core of domain fine-tuning lies in constructing high-quality and large-scale domain fine-tuning data. This embodiment adopts three data generation methods, namely Self-Instruct, Self-QA, and Self-KG, to quickly expand the fine-tuning dataset. By using intelligent algorithms to assist in generating fine-tuning instructions and answers, both the efficiency of data generation is improved, and the quality and diversity of the data are ensured. Therefore, the present invention is based on some existing data + Qwen, and then generates domain-specific fine-tuning data.

[0069] Table 1

[0070] For different existing data, this embodiment adopts three different data generation methods, as shown in Table 1: ① Self-Instruct Method: Self-Instruct is a method for expanding fine-tuning data. In the case of some existing seed fine-tuning data, it can be expanded through Self-Instruct + Qwen to generate more fine-tuning data that relatively meets the requirements. A piece of fine-tuning data includes three parts: instruction, input, and output. The specific process is as follows: First, randomly select some instructions from the seed instructions (manually written instructions / issues accumulated on the business side), and then let Qwen refer to these instructions to generate a series of similar instructions. After obtaining the instructions, let Qwen determine whether this instruction is a "classification" problem or a "generation" problem. Different answer generation strategies will be adopted later. If the problem is a "classification" problem, the "output-first" generation method will be used, that is, first generate the output (specifically which category), and then generate the input based on the instruction and the output. For example, the instruction is: "Judge whether the sentiment of the following sentence is negative or positive". First generate the output category "positive", and then generate the input sentence "I am very happy today" based on the instruction and the category. If the problem is a "generation" problem, the "input-first" generation method will be used, that is, first generate the input, and then generate the output based on the instruction and the input. For example, the instruction is: "Translate the following sentence into English". First generate the input sentence "I am very happy today", and then generate the output answer "I am happy today" based on the instruction and the input. If an instruction does not require an input sentence, the input is empty. For example, the instruction: "What are the exercises for losing weight?".

[0071] Through the above steps, a batch of fine-tuning data can be initially obtained, and further filtering is still required. For example, filter the results with high similarity to the existing data, and filter the obviously low-quality results (instructions are too long or too short). The filtered fine-tuning data can then be continued to be added to the "seed instructions" and loop in this way to continuously generate data.

[0072] ② Self-QA method: For the case where there is no basic seed instruction data, this embodiment adopts the Self-QA method to directly generate instruction data from the document. Through Qwen's analysis and understanding of the unstructured document, instructions and answers related to the document content are generated, thereby constructing an instruction dataset for fine-tuning. The specific process is as follows: First, generate possible instructions through Qwen based on the unstructured document, and then input the instructions and the corresponding document and let Qwen generate the answers to the questions. Here, the document can directly be the document corpus, or unstructured document data can be generated from structured tabular data or graph data.

[0073] Based on the designed Prompt, Qwen can be used to generate instructions and answers respectively, thus constituting instruction fine-tuning data. These data still need to be further filtered by heuristic and rule-based methods to improve the data quality.

[0074] ③Self-KG method: In the case of an existing high-quality knowledge graph, this embodiment adopts the Self-KG method to directly generate instruction data based on the knowledge graph. This method makes full use of the structured information in the knowledge graph to generate instructions and answers closely related to the field of digital governance, providing strong data support for model fine-tuning.

[0075] (4) Fine-tuning method based on P-Tune.

[0076] The method of large model prompt engineering is a relatively easy-to-use way to use large models, but its disadvantages are also very obvious. The necessity of using fine-tuning can be fully reflected through the following analysis: ① Generally, the implementation principle of large models has limitations on the length of the input sequence. The Prompt Engineering method will make the Prompt very long, and the inference cost of large models is positively correlated with the square of the Prompt length. Therefore, the longer the Prompt, the higher the inference cost of the large model.

[0077] ② If the Prompt is too long, it will be truncated due to exceeding the limit, which will cause a significant reduction in the output quality of the large model, resulting in inaccurate output content of the large model; while fine-tuning is different. As long as the fine-tuning training data is of high quality, it can stably output on the dataset or task provided by the project.

[0078] ③ When the effect of Prompt Engineering fails to meet the requirements and the enterprise has relatively good own data, which can better improve the capabilities of the large model in a specific field through its own data, fine-tuning is very applicable at this time. Fine-tuning can enable the project team to provide more data for the LLM than the Prompt, and the model can learn from this data rather than just accessing the data.

[0079] ④ To use the capabilities of large models in personalized services, training a lightweight fine-tuned model for the data of each user is a good solution at this time.

[0080] ⑤ Regarding the issue of data security, if the data cannot be transmitted to third-party large model services, it is very necessary to build one's own large model. Usually, these open-source large models need to be fine-tuned with their own data to meet the business requirements. Fine-tuning of the large model is also required at this time.

[0081] ⑥ There is a certain upper limit to the vectorized matching ability, and there is a bottleneck problem in the matching accuracy for semantic search in search engines.

[0082] ⑦When it comes to introducing external knowledge, in particularly specialized fields, solely relying on vectors, NLP, strategies / rules still doesn't work in certain scenarios; for a specific business, regardless of whether 90% of the problems will be solved, there are only problems that need to be solved or those that don't need to be solved.

[0083] ⑧The advantage of fine-tuning is that it can make the behavior of the LLM adapt to specific nuances, tones, or terms. If we want the model to sound more like a professional in the digital governance industry or use specific industry terms, then fine-tuning on domain-specific data allows us to achieve these customizations.

[0084] ⑨Fine-tuning the LLM to adapt to a specific task or domain depends to a large extent on the quality and quantity of available labeled data. A rich dataset can help the model deeply understand the nuances, complexities, and unique patterns of a specific domain, enabling it to generate more accurate and contextually relevant responses; however, if a limited dataset is used, the improvement brought by fine-tuning may be negligible.

[0085] Essentially, if there is a large amount of labeled data to capture the complexity of the domain, then fine-tuning can provide more customized and refined model behavior; but in the case of limited such data, the RAG (Retrieval-Augmented Generation) system provides a powerful alternative, ensuring that the application remains data-informed and context-aware through its retrieval function. Therefore, for the present invention, massive multi-source multi-modal data is more suitable for fine-tuning the LLM, and the specific implementation includes: 4.1 Introduction of P-Tuning technology: In view of the characteristics of domain-specific data, this embodiment adopts the P-Tuning method for model fine-tuning. Domain-specific data usually contains some specific domain terms, vocabulary, syntactic structures, etc., which are different from general domain data; in this case, fine-tuning the parameters of the model can better adapt to the new data distribution, thereby improving the performance of the model. In contrast, LORA (Layer-wise Relevance Propagation) pays more attention to explaining and understanding the feature weights inside the model, and interprets the results of model predictions by analyzing the model's response to input features. Although LORA can also be applied to the customization process of domain-specific data, it is more suitable for tasks such as model interpretation and feature selection, rather than for fine-tuning models for specific domains. Therefore, for the customization process of domain-specific data, P-Tuning is more suitable, as it pays more attention to the adaptive adjustment of model parameters and can better adapt to the new data distribution and improve model performance.

[0086] 4.2 Continuous Prompt Tuning Techniques: Prompt Tuning, Prefix Tuning, and P-Tuning-V1 are related. These methods are all based on optimizing continuous prompts. Previous work involved manually designing templates or automatically generating templates, collectively referred to as discrete prompts. Discrete prompts have certain limitations. The results obtained may not be optimal, and they are very sensitive to changes in tokens. Therefore, subsequent research directions have also focused on prompts within the continuous space.

[0087] P-Tuning is a soft prompt method for large models, including two versions: P-Tuning only adds new parameters to the Embedding of large models; P-Tuning-V2 adds new parameters before the Embedding and each layer of large models, which is also called deep prompt.

[0088] For many tasks, simply inputting an appropriate prompt at the input end allows large language models to give a relatively reasonable result. However, for manually designed prompts, changes to the prompt are extremely sensitive to the final performance of the model. Adding a word, removing a word, or changing the position can cause significant changes. Therefore, directly let the large model learn a reasonable prompt and then input it into the model to directly obtain a relatively reasonable result. Abandon the previous hard prompt design in the manual or semi-automatic discrete space and adopt the continuous differentiable space soft prompt design to optimize and learn the prompt parameters corresponding to different tasks through end-to-end learning.

[0089] The Prefix Tuning (for the text generation field) method constructs a task-related segment of virtual tokens as a Prefix before the input tokens, and then only updates the parameters of the Prefix part during training while keeping the parameters of other parts in the PLM (Pre-trained Language Model) fixed.

[0090] Prompt Tuning can be regarded as a simplified version of Prefix Tuning. It defines its own Prompt for each task and then concatenates it with the data as input, but only adds prompt tokens at the input layer (adds a fixed-length trainable vector at the input embedding layer and only updates the parameters of this section of the prompt during fine-tuning); in addition, the position of the virtual token is not necessarily the prefix, and the insertion position is optional.

[0091] Regarding the lack of generality in the number of model parameters, previous experiments have proven that P-Tuning has good effects on models with more than 10B parameters and can even achieve the effect of full-scale fine-tuning. However, for medium-scale models, the effects are not very obvious. Its generality for different tasks is also relatively poor. Previous experimental results have shown that it has good effects on some NLU (Natural Language Understanding) tasks and poor effects on relatively difficult tasks such as sequence labeling. The prompt is only added to the embedding layer, which makes the prompt difficult to train and also results in fewer trainable parameters.

[0092] 4.3 Application of P-Tuning-V2: To further improve the fine-tuning effect, P-Tuning-V2 technology is adopted in the following implementation. This technology adds prompt tokens as input to each layer of the large model instead of only adding them at the input layer. This not only increases the number of learnable parameters (from 0.01% of P-Tuning and Prompt Tuning to 0.1% - 3%), but also makes the prompt connected to deeper layers, having a more direct impact on the model output, thereby improving the adaptability and accuracy of the model in the field of digital governance.

[0093] (5)Evaluation and feedback optimization.

[0094] Evaluate the performance of the fine-tuned model on the validation dataset and make necessary adjustments according to the results to optimize the performance of the model. This stage first involves comprehensively and meticulously evaluating the fine-tuned model on a carefully constructed validation dataset. The validation dataset should cover a wide range of representative samples to simulate the diversity in actual application scenarios, thereby ensuring the authenticity and effectiveness of the evaluation results. The selection of evaluation metrics is crucial, and they should be directly related to the actual application requirements of the model, including but not limited to statistical metrics such as accuracy, recall rate, F1 score, precision rate, etc., as well as specific metrics designed for specific tasks. According to the evaluation results, the following optimization work is carried out. The optimization strategies include the following aspects: Parameter Tuning: Use methods such as grid search, random search, or Bayesian optimization to find the optimal combination of model parameters within the preset parameter space to improve model performance.

[0095] Data Augmentation: Increase the diversity of training data by synthesizing new samples, adding noise, data transformation, etc., to help the model learn more robust feature representations.

[0096] Model Structure Adjustment: According to the evaluation feedback, fine-tune or redesign the network structure of the model, such as increasing or decreasing the number of layers, changing the activation function, adjusting the attention mechanism, etc., to improve the model's adaptability to specific tasks.

[0097] Ensemble Learning: Combine the prediction results of multiple models and improve the stability and accuracy of the overall prediction through voting, weighted average, etc.

[0098] As a key link in system maintenance and improvement, the feedback optimization layer is committed to comprehensively collecting user feedback and system operation data to drive continuous optimization and improvement; through user surveys, online evaluation channels, and professional technical support services, this layer can accurately capture user needs and feedback, and insight into potential system problems and improvement directions. At the same time, it monitors the running status and data traffic of the system in real-time, analyzes key performance indicators and abnormal information to ensure the stable operation of the system and quickly respond to potential problems. On this basis, it precisely optimizes the model and algorithm, including but not limited to adjusting model parameters, reconstructing algorithm logic, or introducing cutting-edge technologies to improve system efficiency and accuracy. Finally, keeping up with market trends and technological development, it implements an iterative upgrade strategy for the system, aiming to continuously integrate new functions, optimize the user interaction experience, and improve the overall performance of the system, bringing a more excellent user experience to users.

[0099] (6) Testing and Deployment.

[0100] Evaluate the performance of the model on the test dataset to ensure that it performs well in actual applications, and then proceed with the deployment. The specific operations are as follows: Construct a test dataset: Extract data from data sources that are completely different from the training set and validation set as the test set. Data splitting, manual annotation, etc. can be used to ensure the authenticity and challenging nature of the test set.

[0101] Model Testing: Run the fine-tuned and optimized model on the test set and calculate the values of evaluation metrics to verify the performance of the model in actual applications; when running the fine-tuned and optimized model on the test set, it is necessary to ensure that the test environment is as consistent as possible with the production environment to simulate real operating conditions.

[0102] Performance evaluation: Compare the evaluation results on the test set with the expected goals, and analyze in which aspects the model performs well and which aspects still have room for improvement; When evaluating, pay attention to the comparison benchmark, set one or more benchmark models (such as a simple baseline model, the model of the previous version, etc.), and compare the test results with these benchmark models to evaluate the relative performance of the model.

[0103] Model deployment: Package the model into an API (Application Programming Interface), microservice or integrate it into an existing system to ensure that the model can provide services efficiently and stably. During the deployment process, issues such as the model's operating environment (such as hardware configuration, operating system), installation and configuration of dependent libraries, and the security of the model and data need to be considered.

[0104] In the model deployment process, multi-model fusion (Ensemble Learning) can also be adopted. Multi-model fusion is an ensemble learning technique that combines the prediction results of multiple models to improve the overall prediction performance. In the field of digital intelligence governance, multiple generative AI models with different architectures or different parameters can be trained, and then their prediction results can be processed by weighted average or voting, etc., to obtain more stable and accurate outputs. This method can make full use of the advantages of different models, while reducing the biases and errors that may exist in a single model. By adjusting the weights and fusion strategies of different models, the overall performance of the system can be further optimized to better meet the actual needs of digital intelligence governance.

[0105] Monitoring and maintenance: After deployment, establish a monitoring mechanism to track the model's performance metrics (such as response time, accuracy change) and resource usage (such as CPU, memory occupancy), and at the same time regularly collect new data to update the model to prevent the model from becoming obsolete due to changes in data distribution; In addition, fault troubleshooting and repair work need to be carried out on the model to ensure the continuity and stability of the service.

[0106] (7) Digital intelligence-assisted decision-making.

[0107] The governance auxiliary decision-making layer uses the trained generative AI large model for governance auxiliary decision-making and provides intuitive decision-making support for users. At the governance auxiliary decision-making layer, the present invention deeply integrates the carefully trained and optimized generative AI large models, which, with their powerful data processing capabilities, complex pattern recognition, and creative content generation capabilities, provide unprecedented intellectual support for decision-makers. Through deep learning algorithms, the models can automatically analyze massive and heterogeneous governance data, including but not limited to policy documents, socioeconomic indicators, public feedback, historical cases, etc., to uncover hidden correlations, trends, and potential risk points. Further, these AI models can generate customized and highly accurate decision-making suggestions and predictive analyses based on these data insights and present them to decision-makers in an intuitive and easy-to-understand visual form, such as interactive charts, dynamic simulation scenarios, or directly generate decision-making reports. This intelligent auxiliary decision-making method not only greatly improves the efficiency and accuracy of the decision-making process but also effectively reduces the subjectivity and uncertainty of human judgment, providing strong scientific basis and forward-looking guidance for the governance decisions of enterprises and all sectors of society.

[0108] The above description of the embodiments is to enable those of ordinary skill in the art to understand and apply the present invention. Obviously, those who are familiar with the technology in this field can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative labor. Therefore, the present invention is not limited to the above embodiments, and all improvements and modifications made by those skilled in the art based on the disclosure of the present invention should fall within the protection scope of the present invention.

Claims

1. A digital intelligence governance assistance method based on a generative AI large model, characterized in that, It includes the following steps: (1) Collect a large amount of multi-source heterogeneous data from multiple channels, preprocess this data, and then divide it into a training set, a validation set, and a test set; (2) Select a large language model based on generative AI, and optimize the model according to the specific requirements of the digital governance scenario through fine-tuning techniques; (3) Adjust the structure of the large language model according to the task requirements, and generate vertical domain fine-tuning data; (4) Use the training set data to train the large language model using Prompt Engineering technology, and use the vertical domain fine-tuning data to fine-tune the model during the training process using P-Tuning technology; (5) Input the test set data or real-time data into the trained model for predictive analysis, and provide governance auxiliary decision-making for users according to the predictive analysis results output by the model.

2. The digital intelligence governance assistance method based on the generative AI large model according to claim 1, wherein: The large amount of multi-source heterogeneous data collected in step (1) includes public data, enterprise internal data, social media data, and real-time data generated by Internet of Things devices.

3. The digital governance assistance method based on the generative AI large model according to claim 1, wherein: The preprocessing of the data in step (1) includes four parts: data cleaning, data standardization, data augmentation, and feature engineering. The data cleaning part includes denoising, filling in missing values, and correcting or removing outliers; data standardization includes data type conversion, data encoding, data aggregation, and normalization processing; the feature engineering part includes feature selection and feature extraction. Among them, feature selection uses statistical methods to screen out key features that have an important impact on governance decisions, and feature extraction automatically extracts high-level features from the original data through machine learning methods.

4. The digital intelligence governance assistance method based on the generative AI large model according to claim 1, wherein: The large language model in step (2) uses the GPT series model based on Transformer, which has been pre-trained on a large-scale dataset; in order to inject professional knowledge in the field of digital governance, the fine-tuning technology continues to pre-train on this model, by inputting mixed data, that is, domain data and a small amount of general data, to ensure that the model enhances its understanding and application ability of domain knowledge while maintaining its general ability; then, through SFT and RLHF, further improve the model's understanding and answering ability of domain problems.

5. A digital governance assistance method based on a generative AI large model according to claim 1, characterized in that: The specific method for adjusting the structure of the large language model in step (3) is as follows: For classification tasks, one or more fully connected layers need to be added on top of the pre-trained model. These layers map the hidden representation of the model to the class space required by the task, and use the activation function Softmax to output the prediction probability of each class; For regression tasks, a single fully connected layer is used as the output layer of the model, and the activation function is selected from ReLU, Sigmoid, or linear activation according to the range of the output value to directly output continuous values; For sequence labeling tasks, a CRF layer needs to be added to the model, which takes into account the dependencies between labels to improve the accuracy of labeling; Finally, according to the specific requirements of the task, the input layer and output layer of the model need to be adjusted accordingly. For variable-length input data, additional processing mechanisms including padding and bucketing are introduced to ensure that the model can handle data of different sizes.

6. The digital intelligence governance assistance method based on the generative AI large model according to claim 1, wherein: The specific method for generating vertical domain fine-tuning data in step (3) is as follows: In the case of having some seed instruction data, the Self-Instruct + Qwen method is used for expansion to generate more fine-tuning data that relatively meets the requirements. Each piece of fine-tuning data includes three parts: instruction, input, and output. In the case of having no basic seed instruction data, the Self-QA + Qwen method is used to directly generate instruction data from the document. Qwen analyzes and understands the unstructured document to generate instructions and answers related to the document content, thereby constructing an instruction dataset for fine-tuning. In the case of having a knowledge graph, the Self-KG + Qwen method is used to directly generate instruction data based on the knowledge graph. This method makes full use of the structured information in the knowledge graph to generate instructions and answers closely related to the field of digital governance, providing strong data support for model fine-tuning.

7. The digital intelligence governance assistance method based on the generative AI large model according to claim 1, wherein: The specific method for training the large language model using Prompt Engineering technology in step (4) is as follows: First, design a clear, concise, and targeted Prompt to stimulate the model to generate appropriate output. The Prompt should include clear task instructions, relevant context, example references, and the expected output format. Then, use the validation set and feedback loop method to continuously optimize the effect and performance of the Prompt. Try different Prompt combinations and variants, and select the best Prompt by comparing their performance. Finally, based on the training set data and Prompt, train the large language model in combination with zero-shot learning, few-shot learning, chain-of-thought prompting, and transfer learning methods, and use the Prompt to understand and complete the training task.

8. The digital intelligence governance assistance method based on the generative AI large model according to claim 1, wherein: In step (4), the P-Tuning-V2 technology adds Prompts tokens together with data as input before each layer of the large language model, rather than only adding them to the Embedding layer. The Prompts tokens are a fixed-length trainable vector, and only the parameters of this vector are updated during fine-tuning.

9. The digital intelligence governance assistance method based on the generative AI large model according to claim 1, characterized in that: An intelligent feedback mechanism is incorporated into the model training in step (4). This mechanism collects user feedback information and system operation data comprehensively, analyzes key performance indicators and abnormal information, and precisely optimizes the model and algorithm, including adjusting model parameters, reconstructing algorithm logic, or introducing cutting-edge technologies to improve the efficiency and accuracy of the model.

10. A digital governance assistance method based on a generative AI large model according to claim 1, characterized in that: In step (5), the test set data or real-time data is input into the trained model for predictive analysis. The model automatically learns the inherent laws of the data by virtue of its powerful data processing capabilities, complex pattern recognition and creative content generation capabilities, and mines out hidden correlations, trends and potential risk points, generating customized, highly accurate decision-making recommendations and predictive analysis results, and presenting them to users in the form of interactive charts, dynamic simulation scenarios or directly generated decision reports, so as to provide users with governance support decision-making. At the same time, the knowledge of domain experts is integrated, that is, by writing rule sets, defining logic and constructing domain ontologies, and combining them with the model prediction results, thereby improving the accuracy and reliability of governance support decision-making through logical reasoning and verification.

Images