Method for improving government affair data governance efficiency based on large model technology

Abstract

The invention discloses a method for improving government affair data governance efficiency by relying on a large model technology, and relates to the technical field of data governance, comprising the following steps: aiming at a big data platform, constructing a plurality of agents including data preprocessing, data cataloguing, theme library and thematic library construction, data service development and data analysis; the method comprises the following steps: collecting data from a business system, carrying out standardization processing and storage by using a data preprocessing agent, carrying out automatic data cataloguing by using a data cataloguing agent, generating a data catalogue and a data relation graph, identifying data business attributes by using a theme library and thematic library construction agent, constructing a corresponding library and hooking resources, the data service development agent is used for converting data resources into data services capable of calling APIs by means of natural language interaction, and the data analysis agent is used for carrying out data insight, dynamic generation and flexible display on data directories and data resource indexes. The method is used for government affair data life cycle government and data servitization, and the data government efficiency can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of data governance technology, and specifically to a method for improving the efficiency of government data governance by relying on big model technology. Background Art

[0002] With the continuous development of information technology, the amount of government data has increased dramatically, and the traditional government data governance method can no longer meet the needs. The traditional government data governance method requires a lot of manpower and data customization and development, which is inefficient and cannot respond to business governance needs in an agile manner. Therefore, how to combine the current emerging technologies to improve the efficiency of government data governance has become an urgent problem to be solved.

[0003] First, the sources of government data are extensive and complex. Government data covers the business data of various departments and institutions, including but not limited to administrative approval, public services, law enforcement and supervision. These data come from a wide range of sources and in various formats, including structured data (such as table data in databases), semi-structured data (such as XML, JSON format files) and a large amount of unstructured data (such as documents, images, audio, etc.). For example, a city’s government data may come from different business systems of multiple departments, and the data format and structure of each system are very different.

[0004] Secondly, the government data governance process is cumbersome and inefficient. Traditional government data governance methods include data collection, cleaning, conversion, storage, integration and other links, each of which usually requires manual writing of rules and scripts for processing. In the data cleaning stage, it is necessary to manually identify and process abnormal values ​​and duplicate values; in the data integration stage, it takes a lot of time and effort to match data from different data sources, especially when the amount of data is huge. This traditional governance method is inefficient. Moreover, the data governance processes and standards of different government departments are not completely unified, which further increases the complexity of data governance.

[0005] Existing government data governance solutions mainly rely on open source big data technologies such as Hadoop, Hive, Spark, and Flink to build an integrated government big data platform, with data engineers automatically collecting, processing, analyzing, and applying government data on the big data platform based on user needs. However, current government data governance lacks sufficient intelligent technical support. When faced with massive and complex data, traditional data processing methods have difficulty quickly and accurately identifying patterns and problems in the data, and cannot automatically learn and adapt to changes in the data. For example, report analysis is usually based on a top-down model, where the business side puts forward requirements, developers extract and analyze data, and finally, after a period of development, the results are fed back to the business side.

[0006] In addition, the application of existing large model technology in government data governance is not mature enough and cannot completely replace a large amount of manpower and data customization and development. General large model technology has problems such as weak intention understanding ability, knowledge fragmentation and discreteness, and poor generalization and adaptation ability, resulting in its application scope in government data governance is not wide enough and cannot cover all aspects of government data governance. At the same time, the application cost of existing intelligent agent technology in government data governance is high, requiring a large amount of GPU computing power resources and storage resources, which increases the economic burden of introducing new technology practices in government data governance.

[0007] Therefore, how to improve the application efficiency and effectiveness of intelligent technology in government data governance and reduce application costs has become an urgent problem to be solved. Summary of the invention

[0008] In response to the needs and shortcomings of current technological development, the present invention provides a method for improving the efficiency of government data management by relying on big model technology.

[0009] The present invention provides a method for improving the efficiency of government data management by relying on large model technology, and the technical solution adopted to solve the above technical problems is as follows:

[0010] A method for improving the efficiency of government data governance based on big model technology includes the following steps:

[0011] S1. For the integrated government big data platform, build data preprocessing agents, data cataloging agents, subject library and subject library construction agents, data service development agents, and data analysis agents;

[0012] S2, collect data from the business system, use data preprocessing agents to standardize the collected data to meet quality requirements, and store the data;

[0013] S3, use data cataloging agents to automatically catalog data, generate data directories, and form data relationship maps;

[0014] S4. Use the subject database and thematic database to build an intelligent entity, generate the government subject database and thematic database by identifying the data business attributes, and attach the corresponding data resources to the corresponding database directory;

[0015] S5. Use data services to develop intelligent agents, and through natural language interaction, transform data resources into data services that can be obtained by calling API interfaces;

[0016] S6. Use data analysis agents to gain data insights, dynamically generate, and flexibly display data catalogs and data resource indicators.

[0017] Optionally, the step S1 involved specifically includes:

[0018] Extract various types of government data from the integrated government big data platform, clean, label and classify them, and generate a government knowledge base to make it suitable for model training;

[0019] According to the characteristics of government data, a basic model based on the Transformer architecture is selected;

[0020] Determine the scale parameters and add the specified government affairs module; conduct model training, divide the data set, select the training algorithm and optimizer, use distributed training to improve efficiency, and optimize and iterate based on the results of the validation set; evaluate and optimize after training, select evaluation indicators to measure performance, make targeted improvements, and then compress and optimize the model to form a large government affairs basic model;

[0021] Build a data preprocessing agent based on the government affairs basic big model to remove duplicate data, process missing data, unify format conversion, and integrate data from different sources;

[0022] Building a data cataloging intelligent entity based on the government affairs basic big model to classify, index and describe government affairs data, facilitating data management and retrieval;

[0023] Construct a subject database and topic database construction agent based on the government affairs basic big model, which is used to focus on the government affairs data management of designated topics or departments;

[0024] Build a data service development agent based on the government affairs basic model to provide data services and support online access and monitoring of government affairs data;

[0025] A data analysis intelligent entity is built based on the basic government affairs model to deeply explore the value of government affairs data and display the analysis results through visualization means.

[0026] Optionally, the subject database and special database construction agent involved are applied in government data modeling, including:

[0027] When analyzing government business scenarios, the subject database construction agent uses the government basic big model to learn and understand government business documents and process cases, help sort out business processes and identify key data nodes, so as to accurately determine data needs;

[0028] When designing the data model structure, the subject library and topic library construction intelligent body provides suggestions on data entities and relationships based on its understanding of government data relationships, and optimizes the new data model with reference to the data model experience of similar historical government projects, providing a reference for the design of data association relationships between departments in the local data model.

[0029] Optionally, the subject library and special library construction agent is applied in the definition and maintenance of data quality rules, including:

[0030] The subject database construction agent analyzes massive government data samples, mines potential problems and features in the data, and assists in determining key data quality indicators;

[0031] When writing data quality rules, the subject library and special library construction intelligent agent uses the semantic understanding ability of the government affairs basic big model to convert the rules described in natural language into executable code or rule expressions. In addition, the subject library and special library construction intelligent agent uses the government affairs basic big model to continuously learn new government affairs data specifications and business changes, and promptly proposes rule update suggestions to ensure the timeliness and effectiveness of the rule knowledge base.

[0032] Further optionally, the step S2 specifically includes the following operations:

[0033] S2.1. According to the characteristics of the business system, data real-time requirements and data volume, select a collection method from message queues, log collection, library table exchange and ETL tools, collect data from the business system based on the selected collection method, and convert the collected data into basic data formats;

[0034] S2.2, Data preprocessing agent uses the semantic understanding and deep learning capabilities of the government affairs basic big model to conduct deep semantic analysis on the collected government affairs data, identify the specified data format, special format and government affairs-specific format, and find the same data entity under different expressions by calculating the semantic similarity and matching the government affairs knowledge graph;

[0035] S2.3, Data preprocessing agent relies on in-depth learning and pattern recognition of government language, and uses a large amount of government text training to establish a recognition model. The recognition model can judge the error type based on the context and generate targeted correction suggestions to ensure that the expression after correction conforms to the government context specifications;

[0036] S2.4. In the process of data completion, the data preprocessing agent uses the correlation analysis capability of the government affairs basic big model to perform reasonable data filling based on the internal logical relationship and historical data patterns between government affairs data. In terms of data deduplication, the data preprocessing agent constructs the fingerprint features of government affairs data, compares the similarity between data records, identifies and deletes duplicate data, and retains the key information of the data.

[0037] S2.5. Store the governed data in the data lake based on the integrated government big data platform.

[0038] Further optionally, the step S3 specifically includes the following operations:

[0039] S3.1. When cataloging data, use the data cataloging agent to conduct a comprehensive analysis of the database table, read the table structure and field information, and rely on the government affairs basic big model to deeply understand the data content and business meaning; determine the subject and classification of the data by performing semantic analysis and pattern recognition on government affairs data;

[0040] S3.2. In terms of establishing data relationship graphs, the data cataloging agent uses the natural language understanding capabilities of LLM to automatically extract data entities and one-to-one, one-to-many, and many-to-many relationships between data entities from structured and unstructured data, and presents them in the form of visual graphs; at the same time, with the help of the ontological knowledge and business rules in the government affairs field learned by the basic big model, the data cataloging agent accurately infers the logical connections between data and builds a clear architecture for data integration and utilization;

[0041] S3.3. When data cataloging engineers review the data catalog generated based on the government affairs basic big model, the data cataloging agent serves as an auxiliary tool to display the statistical information of relevant data, including data samples, sources and usage frequency, to help engineers understand the cataloging results of the big model;

[0042] S3.4. In the process of directory data persistence, the government basic big model is deeply integrated with the metadata management system of the integrated government big data platform to ensure that the storage format of the data directory and relationship map conforms to the platform standard, and automatically generates metadata description information of the data directory, including the version information, creation time, and update records of the data directory, so as to achieve automation and standardization of data directory management.

[0043] Further optionally, the step S4 specifically includes the following operations:

[0044] S4.1. The subject database and special database construction agent uses the text classification algorithm in natural language processing, combined with the pre-trained classification model, to perform semantic analysis and topic extraction on the government data content, and realize automatic classification and labeling; S4.2. When defining the subject database and special database based on the subject attributes generated by the subject database and special database construction agent, it provides optimization suggestions for the library table structure and configuration, and provides reasonable solutions for the partitioning and indexing settings of the subject database and special database based on the data distribution characteristics, access patterns and future analysis needs; during the creation process, the subject database and special database construction agent assists in managing the metadata information of the subject database and special database, ensures that the definition information of the library is accurately stored in the configuration management system, and establishes the association relationship between metadata to facilitate subsequent data management and application development;

[0045] S4.3. During the resource attachment process, the subject library and subject library construction agent automatically and accurately attaches the library table resources, file resources, and folder resources to the corresponding subject library or subject library according to the subject, attribute, and label of the data; after the resources are attached, the subject library and subject library construction agent checks the integrity and consistency of the resources to ensure that the resources attached to the subject library or subject library match each other and there is no data conflict or missing;

[0046] S4.4. When checking the rationality of table structure and data quality, the subject database construction agent can automatically discover potential structural problems by comparing the syntax analysis of the table structure, checking the data type and comparing with the reference data model; in terms of data quality detection, in addition to checking the accuracy, completeness and consistency of the data according to the pre-defined data quality rules, the subject database construction agent also uses machine learning algorithms to detect outliers and analyze data distribution to find problems hidden in the data; for the detected problem data, the subject database construction agent conducts in-depth diagnosis and analysis, using data mining technology and government knowledge graphs to trace the root cause of the problem, and provides detailed problem descriptions and solution suggestions;

[0047] S4.5. During the problem analysis phase, the subject database construction intelligent body builds a problem analysis model based on the characteristics of the problem data and the government business process, comprehensively considers the data source, processing links and factors affecting data quality, and provides a comprehensive analysis report for each problem; when rectification suggestions are generated, personalized solutions are provided based on the problem type and severity.

[0048] Further optionally, the step S5 specifically includes the following operations:

[0049] S5.1. During the data service encapsulation process, for secondary development scenarios involving multiple data resources, the data service development agent combines the business requirements described in natural language with its understanding of data structure and business logic to transform complex library, table and file resources into easy-to-call service interfaces.

[0050] S5.2. In terms of integrating data from different sources, the data service development agent conducts in-depth analysis of the situations of each data source, establishes mapping relationships between data and corresponding conversion rules, and achieves seamless data connection.

[0051] Further optionally, the step S6 specifically includes the following operations:

[0052] S6.1. After receiving a question from a user in natural language, the data analysis agent converts the natural language question into an optimized SQL query statement based on its understanding of the data warehouse structure and data distribution, and selects the most effective query path by optimizing query connection conditions and data screening strategies to quickly generate report results.

[0053] S6.2. When displaying chart results, the data analysis agent automatically selects the most appropriate chart type based on data characteristics and user needs, providing users with the most suitable visualization display method.

[0054] A method of the present invention that relies on big model technology to improve the efficiency of government data management has the following beneficial effects compared with the prior art:

[0055] 1. The present invention can conduct government data lifecycle management and data service in an agile and efficient manner, solving the traditional data management implementation mode that is mainly based on human customization and development. Through the data management system that integrates big model technology, relevant departments can quickly respond to data management business demands under limited human resources, significantly improve the data management efficiency and data intelligent analysis capabilities of relevant departments, and support the efficient and compliant use of government data.

[0056] 2. The present invention uses big model technology to perform intelligent processing in the processing links that require a large number of engineers to perform manual maintenance in the system, which can solve the problems of low efficiency and timeliness of existing government data management, achieve efficient, intelligent and accurate management of government data, and provide strong support for decision-making and public services. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Attached Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0058] In order to make the technical solution, the technical problem solved and the technical effect of the present invention more clearly understood, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0059] Embodiment 1:

[0060] Combined with Figure 1 This embodiment proposes a method for improving the efficiency of government data governance by relying on big model technology, which includes the following steps:

[0061] S1. For the integrated government big data platform, build data preprocessing agent, data cataloging agent, subject library and subject library construction agent, data service development agent and data analysis agent. This process specifically includes:

[0062] Extract various types of government data from the integrated government big data platform, clean, label and classify them, and generate a government knowledge base to make it suitable for model training;

[0063] According to the characteristics of government data, a basic model based on the Transformer architecture is selected. This is because the Transformer architecture has demonstrated strong capabilities in fields such as natural language processing and is suitable for processing text and other information that is often involved in government data.

[0064] Determine the scale parameters and add the specified government affairs module; conduct model training, divide the data set, select the training algorithm and optimizer, use distributed training to improve efficiency, and optimize and iterate based on the results of the validation set; evaluate and optimize after training, select evaluation indicators to measure performance, make targeted improvements, and then compress and optimize the model to form a large government affairs basic model;

[0065] Build a data preprocessing agent based on the government affairs basic big model to remove duplicate data, process missing data, unify format conversion, and integrate data from different sources;

[0066] Building a data cataloging intelligent entity based on the government affairs basic big model to classify, index and describe government affairs data, facilitating data management and retrieval;

[0067] Construct a subject database and topic database construction agent based on the government affairs basic big model, which is used to focus on the government affairs data management of designated topics or departments;

[0068] Build a data service development agent based on the government affairs basic model to provide data services and support online access and monitoring of government affairs data;

[0069] A data analysis intelligent entity is built based on the basic government affairs model to deeply explore the value of government affairs data and display the analysis results through visualization means.

[0070] It should be added that the constructed subject database construction agent is applied in government data modeling, including:

[0071] When analyzing government business scenarios, the subject database construction agent uses the government basic big model to learn and understand government business documents and process cases, help sort out business processes and identify key data nodes, so as to accurately determine data needs;

[0072] When designing the data model structure, the subject library and topic library construction intelligent body provides suggestions on data entities and relationships based on its understanding of government data relationships, and optimizes the new data model with reference to the data model experience of similar historical government projects, providing a reference for the design of data association relationships between departments in the local data model.

[0073] The constructed subject database construction agent is applied in the definition and maintenance of data quality rules, including:

[0074] The subject database construction agent analyzes massive government data samples, mines potential problems and features in the data, and assists in determining key data quality indicators;

[0075] When writing data quality rules, the subject library and special library construction intelligent agent uses the semantic understanding ability of the government affairs basic big model to convert the rules described in natural language into executable code or rule expressions. In addition, the subject library and special library construction intelligent agent uses the government affairs basic big model to continuously learn new government affairs data specifications and business changes, and promptly proposes rule update suggestions to ensure the timeliness and effectiveness of the rule knowledge base.

[0076] S2: Collect data from the business system, use the data preprocessing agent to standardize the collected data to meet quality requirements, and store the data. This process includes the following operations:

[0077] S2.1. According to the characteristics of the business system, the real-time requirements of data and the size of data, select a collection method from message queues, log collection, library table exchange and ETL tools, collect data from the business system based on the selected collection method, and convert the collected data into basic data formats; for example, data with high real-time requirements may be more suitable for message queue collection, while data that needs to integrate different data sources may be more suitable for ETL tools; after collecting data, converting it into basic data formats can make subsequent data processing smoother, and a unified data format facilitates various analysis and preprocessing operations, laying the foundation for the overall data governance process;

[0078] S2.2, Data preprocessing agent uses the semantic understanding and deep learning capabilities of the government affairs basic big model to conduct deep semantic analysis on the collected government affairs data, identify the specified data format, special format and government affairs-specific format, and find the same data entity under different expressions by calculating the semantic similarity and matching the government affairs knowledge graph;

[0079] S2.3, Data preprocessing agent relies on in-depth learning and pattern recognition of government language, and uses a large amount of government text training to establish a recognition model. The recognition model can judge the error type based on the context and generate targeted correction suggestions to ensure that the expression after correction conforms to the government context specifications;

[0080] S2.4. In the process of data completion, the data preprocessing agent uses the correlation analysis capability of the government affairs basic big model to perform reasonable data filling based on the internal logical relationship and historical data patterns between government affairs data. In terms of data deduplication, the data preprocessing agent constructs the fingerprint features of government affairs data, compares the similarity between data records, identifies and deletes duplicate data, and retains the key information of the data.

[0081] S2.5. Store the governed data in the data lake based on the integrated government big data platform.

[0082] S3. Use the data cataloging agent to automatically catalog data, generate a data directory, and form a data relationship map. This process includes the following operations:

[0083] S3.1. When cataloging data, use the data cataloging agent to conduct a comprehensive analysis of the database table, read the table structure and field information, and rely on the government affairs basic big model to deeply understand the data content and business meaning; determine the subject and classification of the data by performing semantic analysis and pattern recognition on government affairs data;

[0084] S3.2. In terms of establishing data relationship graphs, the data cataloging agent uses the natural language understanding capabilities of LLM to automatically extract data entities and one-to-one, one-to-many, and many-to-many relationships between data entities from structured and unstructured data, and presents them in the form of visual graphs; at the same time, with the help of the ontological knowledge and business rules in the government affairs field learned by the basic big model, the data cataloging agent accurately infers the logical connections between data and builds a clear architecture for data integration and utilization;

[0085] S3.3. When data cataloging engineers review the data catalog generated based on the government affairs basic big model, the data cataloging agent serves as an auxiliary tool to display the statistical information of relevant data, including data samples, sources and usage frequency, to help engineers understand the cataloging results of the big model;

[0086] S3.4. In the process of directory data persistence, the government basic big model is deeply integrated with the metadata management system of the integrated government big data platform to ensure that the storage format of the data directory and relationship map conforms to the platform standard, and automatically generates metadata description information of the data directory, including the version information, creation time, and update records of the data directory, so as to achieve automation and standardization of data directory management.

[0087] S4. Use the subject database and thematic database to build an intelligent entity. Generate the government subject database and thematic database by identifying the data business attributes, and attach the corresponding data resources to the corresponding database directory. This process includes the following operations:

[0088] S4.1. The construction of subject database and special database The intelligent agent uses the text classification algorithm in natural language processing, combined with the pre-trained classification model, to perform semantic analysis and subject extraction on the government data content, and realize automatic classification and labeling. In this way, government data can be sorted out more efficiently and accurately, so that different data can be classified and managed according to their themes and characteristics, which meets the actual needs of government data classification management.

[0089] S4.2, when defining thematic libraries and thematic libraries based on thematic attributes generated by thematic and thematic library construction agent, provide optimization suggestions for the library table structure and configuration, and provide reasonable solutions for the partitioning and indexing settings of thematic and thematic libraries based on the data distribution characteristics, access patterns and future analysis needs; during the creation process, thematic and thematic library construction agent assists in managing the metadata information of thematic and thematic libraries, ensuring that the library definition information is accurately stored in the configuration management system, and establishing the association relationship between the metadata, which is a very critical basic link for subsequent standardized data management and application development work, and can make the entire data management process more orderly and efficient;

[0090] S4.3. During the resource attachment process, the subject library and subject library construction agent automatically and accurately attaches the library table resources, file resources, and folder resources to the corresponding subject library or subject library according to the subject, attribute, and label of the data; after the resources are attached, the subject library and subject library construction agent checks the integrity and consistency of the resources to ensure that the resources attached to the subject library or subject library match each other, avoid data usage problems caused by resource mismatch, data conflict, or missing, and effectively maintain the quality and availability of data resources in the subject library and subject library;

[0091] S4.4. When checking the rationality of table structure and data quality, the subject database construction agent can automatically discover potential structural problems by comparing the syntax analysis of the table structure, checking the data type and comparing with the reference data model; in terms of data quality detection, in addition to checking the accuracy, completeness and consistency of the data according to the pre-defined data quality rules, the subject database construction agent also uses machine learning algorithms to detect outliers and analyze data distribution to find problems hidden in the data; for the detected problem data, the subject database construction agent conducts in-depth diagnosis and analysis, using data mining technology and government knowledge graphs to trace the root cause of the problem, and provides detailed problem descriptions and solution suggestions. This series of operations helps to deeply explore the reasons behind data problems so as to fundamentally solve the problem and improve the overall data quality;

[0092] S4.5. During the problem analysis phase, the subject database construction intelligent body builds a problem analysis model based on the characteristics of the problem data and the government business process, comprehensively considers the data source, processing links and factors affecting data quality, and provides a comprehensive analysis report for each problem; when rectification suggestions are generated, personalized solutions are provided based on the problem type and severity.

[0093] It should be added that, usually, the pre-training process of the classification model includes the following steps: ① Collect a large amount of government-related text data as pre-training materials. These data can cover various government documents, reports, policies and regulations, etc., and the collected data must be cleaned, labeled and other pre-processed, such as removing noise data and labeling the text according to certain classification standards (for example, according to different government themes, business fields, etc.) to ensure data quality and availability; ② According to the characteristics of government data and the classification task goals to be achieved, select a suitable pre-training model architecture, such as adjusting and applying the Transformer architecture and other infrastructure that performs well in natural language processing; the pre-training goal can be to learn the semantic representation, lexical relations, common topic patterns, etc. in government texts, such as allowing the model to master the relationship between different terms in the government field and the language characteristics of texts on different topics through pre-training; ③ Use a suitable training algorithm (such as the common stochastic gradient descent and and its variants, etc.), and conduct multiple rounds of training on large-scale annotated government data; during the training process, the model will continuously adjust its own parameters to learn various rules and characteristics in government text data, such as how to judge the government subject category to which it belongs based on the words, word order and other information in the sentence; at the same time, some hyperparameters (such as learning rate, batch size, etc.) may be set and adjusted and optimized according to the verification situation during the training process to ensure that the pre-training process is stable and develops in the direction of improving model performance; ④ Use a part of the reserved verification data set to evaluate the model at different stages of the pre-training process, and observe the performance of the model on these verification data. The performance of indicators such as classification accuracy, recall rate, F1 value, etc., is used to judge the learning effect of the model and whether there are problems such as overfitting; according to the evaluation results, the model's hyperparameters or training strategies are further adjusted to continuously optimize the model performance until a relatively satisfactory pre-training effect is achieved, so that the model can play a good role in subsequent applications in government data classification tasks.

[0094] S5. Use data services to develop intelligent agents. Through natural language interaction, data resources are converted into data services that can be obtained by calling API interfaces. This process includes the following operations:

[0095] During the data service encapsulation process, for secondary development scenarios involving multiple data resources, the data service development agent combines the business requirements described in natural language with its understanding of data structure and business logic to transform complex library, table and file resources into easy-to-call service interfaces.

[0096] S5.2. In terms of integrating data from different sources, the data service development agent conducts in-depth analysis of the situations of each data source, establishes mapping relationships between data and corresponding conversion rules, and achieves seamless data connection.

[0097] S6. Use data analysis agents to conduct data insights, dynamic generation, and flexible display of data catalogs and data resource indicators. This process includes the following operations:

[0098] S6.1. After receiving a question from a user in natural language, the data analysis agent converts the natural language question into an optimized SQL query statement based on its understanding of the data warehouse structure and data distribution, and selects the most effective query path by optimizing query connection conditions and data screening strategies to quickly generate report results.

[0099] S6.2. When displaying chart results, the data analysis agent automatically selects the most appropriate chart type based on data characteristics and user needs, providing users with the most suitable visualization display method.

[0100] In summary, the method of the present invention that relies on big model technology to improve the efficiency of government data governance can agilely and efficiently carry out government data lifecycle governance and data service, which solves the implementation mode of traditional data governance that is mainly based on human customized development. It can significantly improve the data governance efficiency and data intelligent analysis capabilities of relevant departments, realize efficient, intelligent and accurate governance of government data, and provide strong support for decision-making and public services.

[0101] The above specific examples are used to explain the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by technicians in this technical field without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A method for improving the efficiency of government data governance based on big model technology, characterized in that: The steps include: S1. For the integrated government big data platform, build data preprocessing agents, data cataloging agents, subject library and subject library construction agents, data service development agents, and data analysis agents; S2, collect data from the business system, use data preprocessing agents to standardize the collected data to meet quality requirements, and store the data; S3, use data cataloging agents to automatically catalog data, generate data directories, and form data relationship maps; S4. Use the subject database and thematic database to build an intelligent entity, generate the government subject database and thematic database by identifying the data business attributes, and attach the corresponding data resources to the corresponding database directory; S5. Use data services to develop intelligent agents, and through natural language interaction, transform data resources into data services that can be obtained by calling API interfaces; S6. Use data analysis agents to gain data insights, dynamically generate, and flexibly display data catalogs and data resource indicators.

2. According to claim 1, a method for improving the efficiency of government data management by relying on big model technology is characterized in that: The step S1 specifically includes: Extract various types of government data from the integrated government big data platform, clean, label and classify them, and generate a government knowledge base to make it suitable for model training; According to the characteristics of government data, a basic model based on the Transformer architecture is selected; Determine the scale parameters and add the specified government affairs module; conduct model training, divide the data set, select the training algorithm and optimizer, use distributed training to improve efficiency, and optimize and iterate based on the results of the validation set; evaluate and optimize after training, select evaluation indicators to measure performance, make targeted improvements, and then compress and optimize the model to form a large government affairs basic model; Build a data preprocessing agent based on the government affairs basic big model to remove duplicate data, process missing data, unify format conversion, and integrate data from different sources; Building a data cataloging intelligent entity based on the government affairs basic big model to classify, index and describe government affairs data, facilitating data management and retrieval; Construct a subject database and topic database construction agent based on the government affairs basic big model, which is used to focus on the government affairs data management of designated topics or departments; Build a data service development agent based on the government affairs basic model to provide data services and support online access and monitoring of government affairs data; A data analysis intelligent entity is built based on the basic government affairs model to deeply explore the value of government affairs data and display the analysis results through visualization means.

3. According to claim 2, a method for improving the efficiency of government data management by relying on big model technology is characterized in that: The subject database and special database construction agent is applied in government data modeling, specifically including: When analyzing government business scenarios, the subject database construction agent uses the government basic big model to learn and understand government business documents and process cases, help sort out business processes and identify key data nodes, so as to accurately determine data needs; When designing the data model structure, the subject library and topic library construction intelligent body provides suggestions on data entities and relationships based on its understanding of government data relationships, and optimizes the new data model with reference to the data model experience of similar historical government projects, providing a reference for the design of data association relationships between departments in the local data model.

4. According to claim 3, a method for improving the efficiency of government data management by relying on big model technology is characterized in that: The subject database and special database construction agent is applied in the definition and maintenance of data quality rules, specifically including: The subject database construction agent analyzes massive government data samples, mines potential problems and features in the data, and assists in determining key data quality indicators; When writing data quality rules, the subject library and special library construction intelligent agent uses the semantic understanding ability of the government affairs basic big model to convert the rules described in natural language into executable code or rule expressions. In addition, the subject library and special library construction intelligent agent uses the government affairs basic big model to continuously learn new government affairs data specifications and business changes, and promptly proposes rule update suggestions to ensure the timeliness and effectiveness of the rule knowledge base.

5. According to claim 2, a method for improving the efficiency of government data management by relying on big model technology is characterized in that: The step S2 specifically includes the following operations: S2.

1. According to the characteristics of the business system, data real-time requirements and data volume, select a collection method from message queues, log collection, library table exchange and ETL tools, collect data from the business system based on the selected collection method, and convert the collected data into basic data formats; S2.2, Data preprocessing agent uses the semantic understanding and deep learning capabilities of the government affairs basic big model to conduct deep semantic analysis on the collected government affairs data, identify the specified data format, special format and government affairs-specific format, and find the same data entity under different expressions by calculating the semantic similarity and matching the government affairs knowledge graph; S2.3, Data preprocessing agent relies on in-depth learning and pattern recognition of government language, and uses a large amount of government text training to establish a recognition model. The recognition model can judge the error type based on the context and generate targeted correction suggestions to ensure that the expression after correction conforms to the government context specifications; S2.

4. In the process of data completion, the data preprocessing agent uses the correlation analysis capability of the government affairs basic big model on government affairs data, and performs reasonable data filling based on the internal logical relationship and historical data patterns between government affairs data. In terms of data deduplication, the data preprocessing agent constructs the fingerprint features of government affairs data, compares the similarity between data records, identifies and deletes duplicate data, and retains the key information of the data. S2.

5. Store the governed data in the data lake based on the integrated government big data platform.

6. According to claim 2, a method for improving the efficiency of government data management by relying on big model technology is characterized in that: The step S3 specifically includes the following operations: S3.

1. When cataloging data, use the data cataloging agent to conduct a comprehensive analysis of the database table, read the table structure and field information, and rely on the government affairs basic big model to deeply understand the data content and business meaning; determine the subject and classification of the data by performing semantic analysis and pattern recognition on government affairs data; S3.

2. In terms of establishing data relationship graphs, the data cataloging agent uses the natural language understanding capabilities of LLM to automatically extract data entities and one-to-one, one-to-many, and many-to-many relationships between data entities from structured and unstructured data, and presents them in the form of visual graphs. At the same time, with the help of the ontological knowledge and business rules in the government affairs field learned by the basic big model, the data cataloging agent accurately infers the logical connections between data and builds a clear architecture for data integration and utilization. S3.

3. When data cataloging engineers review the data catalog generated based on the government affairs basic big model, the data cataloging agent serves as an auxiliary tool to display the statistical information of relevant data, including data samples, sources and usage frequency, to help engineers understand the cataloging results of the big model; S3.

4. In the process of directory data persistence, the government basic big model is deeply integrated with the metadata management system of the integrated government big data platform to ensure that the storage format of the data directory and relationship map conforms to the platform standard, and automatically generates metadata description information of the data directory, including the version information, creation time, and update records of the data directory, so as to achieve automation and standardization of data directory management.

7. According to claim 2, a method for improving government data management efficiency by relying on big model technology is characterized in that: The step S4 specifically includes the following operations: S4.

1. The subject database and special database construction agent uses the text classification algorithm in natural language processing, combined with the pre-trained classification model, to perform semantic analysis and topic extraction on the government data content, and realize automatic classification and labeling; S4.

2. When defining the subject database and special database based on the subject attributes generated by the subject database and special database construction agent, it provides optimization suggestions for the library table structure and configuration, and provides reasonable solutions for the partitioning and indexing settings of the subject database and special database based on the data distribution characteristics, access patterns and future analysis needs; During the creation process, the subject library and subject library construction agent assists in managing the metadata information of the subject library and subject library, ensures that the definition information of the library is accurately stored in the configuration management system, and establishes the association relationship between the metadata to facilitate subsequent data management and application development; S4.

3. During the resource attachment process, the subject library and subject library construction agent automatically and accurately attaches the library table resources, file resources, and folder resources to the corresponding subject library or subject library according to the subject, attribute, and label of the data; after the resources are attached, the subject library and subject library construction agent checks the integrity and consistency of the resources to ensure that the resources attached to the subject library or subject library match each other and there is no data conflict or missing; S4.

4. When checking the rationality of table structure and data quality, the subject database construction agent can automatically discover potential structural problems by comparing the syntax analysis of the table structure, checking the data type and comparing with the reference data model; in terms of data quality detection, in addition to checking the accuracy, completeness and consistency of the data according to the pre-defined data quality rules, the subject database construction agent also uses machine learning algorithms to detect outliers and analyze data distribution to find problems hidden in the data; for the detected problem data, the subject database construction agent conducts in-depth diagnosis and analysis, using data mining technology and government knowledge graphs to trace the root cause of the problem, and provides detailed problem descriptions and solution suggestions; S4.

5. During the problem analysis phase, the subject database construction intelligent body builds a problem analysis model based on the characteristics of the problem data and the government business process, comprehensively considers the data source, processing links and factors affecting data quality, and provides a comprehensive analysis report for each problem; when rectification suggestions are generated, personalized solutions are provided based on the problem type and severity.

8. According to claim 2, a method for improving the efficiency of government data management by relying on big model technology is characterized in that: The step S5 specifically includes the following operations: S5.

1. During the data service encapsulation process, for secondary development scenarios involving multiple data resources, the data service development agent combines the business requirements described in natural language with its understanding of data structure and business logic to transform complex library, table and file resources into easy-to-call service interfaces. S5.

2. In terms of integrating data from different sources, the data service development agent conducts in-depth analysis of the situations of each data source, establishes mapping relationships between data and corresponding conversion rules, and achieves seamless data connection.

9. According to claim 2, a method for improving the efficiency of government data management by relying on big model technology is characterized in that: The step S6 specifically includes the following operations: S6.

1. After receiving a question from a user in natural language, the data analysis agent converts the natural language question into an optimized SQL query statement based on its understanding of the data warehouse structure and data distribution, and selects the most effective query path by optimizing query connection conditions and data screening strategies to quickly generate report results. S6.

2. When displaying chart results, the data analysis agent automatically selects the most appropriate chart type based on data characteristics and user needs, providing users with the most suitable visualization display method.

Images