A system and method for automatically labeling government big data

Abstract

The present invention discloses a system and method for automatic labeling of government big data, and relates to the field of data processing technology. The system includes a data source management module, a label management module, a task management module, an execution node management module, a task execution node and a label verification module; the data source management module is used to obtain data to be labeled from government big data; the label management module is used to create or modify label configuration information according to the data to be labeled; the task management module is used to generate task instructions; the execution node management module is used to determine the task execution node from multiple service nodes; the task execution node is used to label the data to be labeled with corresponding labels according to the labeling rules; the label verification module is used to verify the accuracy of the labeling results according to the mutual relationship between the labeling rules. The present invention can automatically label and verify a large amount of government data, reduce labor costs, and enhance the consistency and accuracy of data labeling.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a system and method for automatically labeling government big data. Background Art

[0002] Government big data refers to a large amount of data collected, generated and used by government management agencies in their daily operations. These data usually include citizen information, corporate registration information, public safety records, economic statistics, industrial and commercial information, etc. They are of high value, high dimension and high complexity, and are of great significance to government agency decision-making, service optimization and policy formulation. The management and analysis of government big data is crucial to improving the transparency, efficiency and quality of government agencies' work, and also puts forward higher requirements for ensuring data security and privacy protection.

[0003] With the deepening of digital transformation, government big data is accumulating at an unprecedented rate, and its scale and diversity are increasing. This has brought new challenges to the management and analysis of government big data. In order to improve the management efficiency of government big data, it is necessary to label government big data.

[0004] However, the current government big data labeling work often relies on manual data labeling methods, which not only has limited efficiency and is difficult to cope with the growing amount of data, but is also costly and easily affected by subjectivity, making it difficult to ensure the accuracy and consistency of the labeling results. Summary of the invention

[0005] In view of the above-mentioned technical problems and defects, the purpose of the present invention is to provide a system and method for automatic labeling of government big data, which can automatically label a large amount of government data and verify the labeling results, thereby improving the efficiency of data labeling processing, reducing labor costs, and enhancing the consistency and accuracy of data labeling.

[0006] To achieve the above-mentioned purpose, in a first aspect, the present invention provides an automatic labeling system for government big data, comprising a data source management module, a label management module, a task management module, an execution node management module, a task execution node and a label verification module; the data source management module is used to obtain data to be labeled from government big data; the label management module is used to create or modify label configuration information according to the data to be labeled, and the label configuration information includes label classification and labeling rules; the task management module is used to create a labeling task according to the data to be labeled and the label configuration information, and generate a task instruction; the execution node management module is used to determine a task execution node from multiple service nodes after receiving the task instruction; the task execution node is used to execute the labeling task, so as to label the data to be labeled with the corresponding label classification according to the labeling rule, and obtain the labeling result; the label verification module is used to verify the accuracy of the labeling result according to the relationship between the labeling rules.

[0007] The above-mentioned automatic labeling system for government big data of the present invention realizes automation and intelligence of data processing through integrated multiple modules, which significantly improves the efficiency and accuracy of government data labeling. The data source management module ensures the efficient acquisition of data, the flexibility of the label management module allows rapid adaptation to new data features and business needs, and the task management module optimizes resource allocation through intelligent task creation and scheduling. The dynamic task allocation mechanism of the execution node management module ensures the rapid response and processing of tasks, and the precise execution of the task execution node directly produces preliminary labeling results. Finally, the label verification module automatically verifies the labeling results by deeply analyzing the complex relationship between the labeling rules, ensuring high-quality output of data labeling. The present invention can automatically label a large amount of government data and verify the labeling results, improve the efficiency and reliability of data labeling processing, reduce manual labeling operations, reduce labor costs, and enhance the consistency and accuracy of data labeling.

[0008] In some embodiments, the label verification module is further used to adjust the sensitivity of the verification strategy through a logistic regression model according to the verification results.

[0009] By adopting the technical solution of this embodiment, the label verification module realizes dynamic adjustment of the sensitivity of the verification strategy through the application of the logistic regression model to respond to the changing data characteristics and labeling requirements. This adaptive adjustment mechanism enables the system to optimize the model parameters according to the actual verification results, so as to more accurately identify and correct labeling errors. With the passage of time and the accumulation of data, the system improves the accuracy of verification through continuous learning and optimization, reduces false positives and false negatives, and ensures the high accuracy and reliability of the government big data labeling results.

[0010] In some embodiments, the logistic regression model may include the following formula:

[0011] ;

[0012] Among them, y is the target variable, representing the verification result; p(y=1|x) represents the probability that y=1 when the feature x is given; α is the intercept term of the model, and β is the coefficient of the model.

[0013] By adopting the technical solution of this embodiment, based on the formula in the logistic regression model, the system can more accurately estimate the probability of correctness of each annotation, thereby adjusting the verification rules and improving the accuracy and efficiency of the system. This probability-based verification method enables the system to flexibly adjust when facing complex and changeable data, ensuring the quality of data annotation, while reducing false positives and false negatives, and improving the overall reliability of government big data annotation.

[0014] In some embodiments, the execution node management module is specifically used to monitor the online status and load condition of the service node in real time, and determine the task execution node according to the online status and the load condition.

[0015] By adopting the technical solution of this embodiment, the efficient operation of the government big data automatic annotation system and the optimal allocation of resources are ensured by executing the real-time monitoring function of the node management module. By real-time monitoring of the online status and load of the service nodes, the system can dynamically adjust the task allocation to avoid overload and ensure rapid task response. This intelligent load balancing strategy improves the stability and reliability of the system, while increasing the throughput of data processing and ensuring the continuity and efficiency of the annotation tasks.

[0016] In some embodiments, the service node is deployed in a distributed manner in multiple servers, and the execution node management module is further used to provide registration services for the service node after the service node is deployed.

[0017] The technical solution of this embodiment provides a high degree of scalability and flexibility for the automatic annotation system of government big data through distributed deployment of service nodes and registration services. By deploying service nodes on multiple servers, the system can handle larger data sets, while the registration service ensures the rapid integration and unified management of new nodes. This distributed architecture enhances the fault tolerance of the system. Even if some nodes fail, it does not affect the execution of the entire annotation task, ensuring the continuity and stability of government data annotation.

[0018] In some embodiments, the government big data automatic labeling system also includes a task execution recording module and an information display module. The task execution recording module is used to record the task execution status of the labeling task, and the information display module is used to receive the task execution status and display the task execution status to the user.

[0019] The technical solution of this embodiment provides users with comprehensive labeling task monitoring and display capabilities through the task execution recording module and the information display module. The recording module records each link of the task execution in detail, while the display module presents this information to the user in an intuitive way, allowing the user to understand the task progress and execution status in real time. This transparency not only enhances the user's trust in the system, but also provides strong support for system operation and maintenance and task management.

[0020] In some embodiments, the task execution node is further used to transmit the obtained annotation results to the result database for storage after executing the annotation task.

[0021] By adopting the technical solution of this embodiment, the task execution node has the function of transmitting the annotation results to the result database for storage, which provides the government big data automatic annotation system with the ability to persist and reuse data. This centralized storage of annotation results facilitates subsequent data analysis, auditing and reprocessing, ensuring the long-term value and traceability of the data. At the same time, this also facilitates the sharing and exchange of data, and promotes the openness and interconnection of government data.

[0022] In some embodiments, the government big data automatic annotation system also includes a report generation module, which is used to generate a government big data annotation report based on task execution status and annotation results through a template engine and natural language technology.

[0023] By adopting the technical solution of this embodiment, the report generation module automatically generates annotation reports through the template engine and natural language technology, which greatly improves the report generation efficiency and quality of the government big data automatic annotation system. The module can quickly produce structured and information-rich reports based on the task execution status and annotation results, reducing the workload of manual report writing while ensuring the consistency and professionalism of the reports. The generation and distribution of automated reports provides timely and accurate data support for government decision makers.

[0024] In some embodiments, the data source management module is further used to associate and aggregate the data to be labeled according to data identification identifiers through a data exchange platform.

[0025] By adopting the technical solution of this embodiment, the data source management module associates and aggregates data through the data exchange platform, providing a powerful data integration capability for the government big data automatic annotation system. This aggregation method based on data identification not only improves the organization and availability of data, but also makes it possible for cross-departmental and cross-system data sharing and collaborative work. This is of great significance for breaking information silos and realizing centralized management and efficient use of government data.

[0026] In a second aspect, the present invention provides a method for automatic labeling of government big data, which is applied to the automatic labeling system for government big data provided by any of the above-mentioned embodiments, the method comprising: obtaining data to be labeled from government big data; creating or modifying label configuration information based on the data to be labeled, the label configuration information including label classification and labeling rules; creating a labeling task based on the data to be labeled and the label configuration information, and generating a task instruction; after receiving the task instruction, determining a task execution node from multiple service nodes; executing the labeling task through the task execution node, so as to label the data to be labeled with the corresponding label classification according to the labeling rule, and obtain a labeling result; and verifying the accuracy of the labeling result based on the relationship between the labeling rules.

[0027] The technical effects of the method provided in the second aspect of the present invention refer to the automatic labeling system provided in the first aspect and will not be described in detail here.

[0028] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:

[0029] 1. Highly automated data processing flow: The government big data automatic labeling system of the present invention realizes a high degree of automation of the data processing flow. From data acquisition in the data source management module, intelligent label configuration in the label management module, automated task generation in the task management module, to dynamic task allocation and execution in the execution node management module, the entire system reduces manual intervention and improves the speed and scale of data processing. This automation not only improves efficiency, but also enhances the accuracy and consistency of data labeling by reducing human errors.

[0030] 2. Enhanced data security and quality control: By introducing the label verification module, the present invention enhances data security and quality control. The label verification module verifies the accuracy of labels based on the logical relationship between the labeling rules, ensuring high-quality data output. In addition, the use of the task execution record module and the information display module improves the transparency of system operations, making the data labeling process traceable and auditable, further enhancing the reliability of the data and the trustworthiness of the system.

[0031] 3. Intelligent resource optimization and load balancing: The present invention implements resource optimization and load balancing by executing the node management module to monitor and intelligently schedule the service nodes in real time. The distributed deployment and registration service of the service nodes, combined with the function of real-time monitoring of the online status and load status, ensures that tasks can be reasonably allocated according to the current load and capacity of each node. This intelligent scheduling mechanism improves the overall performance of the system, ensures the efficiency and stability of task execution, and maintains the system's response speed and processing power even under high load conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into and constitute a part of the specification, showing embodiments consistent with the present invention, and together with the specification, are used to explain the principles of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0033] Figure 1 It is a schematic diagram of the architecture of an automatic annotation system for government big data according to an embodiment of the present invention;

[0034] Figure 2 This is a schematic diagram of the architecture of another automatic annotation system for government big data according to an embodiment of the present invention;

[0035] Figure 3 It is a flow chart of a method for automatic annotation of government big data according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The terms used in the following embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to be limiting of the present invention. As used in the specification of the present invention, the singular expressions "a", "a", "above", "the" and "this" are intended to also include plural expressions, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present invention refers to any or all possible combinations comprising one or more of the listed items.

[0037] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, unless otherwise specified, the meaning of "plurality" is two or more.

[0038] It should also be noted that, unless otherwise clearly specified and limited, in the embodiments of the present invention, the terms such as "setting" and "connection" should be understood in a broad sense. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two components; it can be a wired communication connection or a wireless communication connection. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. The embodiments of the present invention are described in detail below.

[0039] Taking enterprise management as an example, the process of manually labeling government big data in related technologies usually includes the following steps: First, staff need to extract relevant information from structured and unstructured data from different sources such as enterprise registration information, tax records, and market supervision reports. Then, they manually classify and label this information according to the business rules and needs of government agencies, such as "industry category", "enterprise size", "credit rating", etc. In this process, staff may need to read a large number of text reports, review various documents submitted by enterprises, and even need to judge the compliance of enterprises based on specific legal or policy standards.

[0040] This manual labeling is not only inefficient, but also easily affected by the subjective judgment of staff, resulting in inconsistency and errors in the labeling results. In addition, with the increase in the number of enterprises and the accumulation of data, the workload and complexity of manual labeling are also increasing, which brings challenges to data analysis and decision support of government agencies.

[0041] Compared with the above-mentioned manual labeling, this embodiment proposes an automatic labeling system for government big data, which can improve the speed and scale of data processing, while reducing errors and inconsistencies caused by human factors. The system can continuously process and analyze large amounts of data, automatically identify government big data information, accurately classify and apply preset labels, such as enterprise type, scale, credit rating, etc., and verify the labeling results after labeling, thereby greatly improving the labeling efficiency and accuracy. In addition, the system has the ability to self-learn and optimize. As time goes by and data accumulates, the accuracy and efficiency of labeling will continue to improve. The automated process also reduces dependence on human resources, reduces long-term operating costs, and enhances the flexibility and scalability of the system, enabling it to adapt to the ever-changing government needs and policy environment.

[0042] The present invention provides a government affairs big data automatic labeling system, such as Figure 1 As shown, the system includes a data source management module 101 , a label management module 102 , a task management module 103 , an execution node management module 104 , a task execution node 105 and a label verification module 106 .

[0043] The data source management module 101 is used to obtain data to be labeled from government big data.

[0044] In this embodiment, the data source management module 101 is the data input core of the government big data automatic annotation system, and is responsible for efficiently and accurately obtaining the data to be annotated from the government big data. This module can seamlessly connect with structured data sources such as MySQL, Oracle, and unstructured data sources such as text files, PDF documents, etc. by integrating multiple database drivers and data interfaces. It has the functions of data extraction, synchronization and preprocessing to ensure the quality and consistency of data.

[0045] For example, in the field of urban management, this module can extract video stream data from the city monitoring system in real time, or obtain vehicle violation record data from the traffic management department, providing raw materials for subsequent automatic labeling work.

[0046] The tag management module 102 is used to create or modify tag configuration information according to the data to be labeled, wherein the tag configuration information includes tag classification and labeling rules.

[0047] In this embodiment, the tag management module 102 can automatically analyze the characteristics and content of the data to be labeled and intelligently identify key information and patterns in the data through integrated machine learning algorithms and natural language processing technology. Based on these analysis results, the tag configuration information can be automatically created or updated, including defining new tag categories and setting corresponding tagging rules.

[0048] For example, if new enterprise registration information is encountered in the system, the label management module 102 can automatically identify the industry and scale of the enterprise, and generate or adjust the corresponding label without human intervention. In addition, the module can continuously optimize and refine the label system according to the statistical laws and correlations in the data to ensure the accuracy and timeliness of the labels, thereby realizing an automated label management process without manual operation. This intelligent label management not only improves efficiency and reduces human errors, but also enables the system to adapt to rapidly changing data environments and business needs.

[0049] On the other hand, the tag management module 102 also allows users to dynamically create or modify tag configuration information based on the characteristics of the data to be labeled and business needs. This process usually involves defining new tag categories, such as "enterprise type" or "policy area", and setting specific tagging rules for these categories. Users can enter rules through an intuitive interface, for example, automatically classify the size of an enterprise based on its annual revenue and number of employees, or automatically mark policy issues based on keywords in the text content. In addition, the tag management module 102 also supports the adjustment of existing tags and rules to adapt to policy changes or the evolution of data characteristics. For example, if new regulations require additional classification of enterprises, users can add new tags in the tag management module 102 and define the corresponding tagging logic. The system will then classify and tag the new data to be labeled according to these updated rules. The flexibility and configurability of this module ensure that the tagging process accurately reflects the latest business needs and policy orientations.

[0050] The task management module 103 is used to create a labeling task according to the data to be labeled and the label configuration information, and generate a task instruction.

[0051] In this embodiment, the task management module 103 is the intelligent task planning and allocation center in the government big data automatic labeling system. It automatically designs and generates labeling tasks by analyzing the data set to be labeled and the existing label configuration information.

[0052] The task management module 103 first evaluates the characteristics and requirements of the data set, and then intelligently constructs the task parameters and execution process according to the label classification and labeling rules provided by the label management module 102. Next, the task management module 103 encapsulates this information into specific task instructions, including specific requirements, priorities, expected completion time, etc. for data labeling. These task instructions are then automatically assigned to the execution node management module 104 to ensure that each labeling task can be started and executed efficiently and accurately.

[0053] For example, if the data to be labeled is a batch of new enterprise registration records, the task management module 103 will create a labeling task based on the enterprise size and industry classification rules defined in the label configuration information, automatically generate task instructions containing these rules, and prepare to distribute the task to the appropriate execution node for processing. This automated process not only improves the efficiency of task creation, but also ensures the rationality of task allocation and consistency of execution.

[0054] The execution node management module 104 is used to determine a task execution node 105 from a plurality of service nodes after receiving the task instruction.

[0055] In this embodiment, the execution node management module 104 plays the role of task allocation and resource optimization in the government big data automatic annotation system. After receiving the task instruction from the task management module 103, the module first evaluates the status of each service node in the current system, including their workload, performance indicators and availability.

[0056] Then, based on this information and the specific requirements of the task, the execution node management module 104 uses built-in intelligent scheduling algorithms, such as polling, least connection or priority queue, to select the most suitable node from multiple service nodes to execute the task, and the selected service node is the task execution node 105. During the selection process, the module may also consider factors such as the complexity of the task, the sensitivity of the data, the geographical location of the execution node, and the expected execution time to ensure that the task can be completed efficiently, stably and safely.

[0057] For example, if a labeling task needs to process a large amount of data and has high requirements for processing speed, the execution node management module 104 may select a service node with strong computing power and low current load to execute the task, thereby achieving optimal resource allocation and high efficiency of task execution.

[0058] The task execution node 105 is used to execute the labeling task, so as to label the data to be labeled with corresponding label classifications according to the labeling rules to obtain a labeling result.

[0059] The labeling results include the label classifications assigned to data items, which reflect the characteristics, attributes or classifications of the data. The labeling results make the originally unstructured or semi-structured data orderly and searchable, facilitating the further analysis, storage, retrieval and management of the data, and are an indispensable information basis for application scenarios such as government decision-making, policy formulation and resource optimization.

[0060] In this embodiment, the task execution node 105 is an execution unit in the government big data automatic annotation system, which is responsible for actually annotating the data according to the task instruction and annotation rules. When the task execution node 105 receives the task instruction from the execution node management module 104, it will first parse the specific requirements and annotation rules of the task.

[0061] Subsequently, the automated labeling engine on the node will start, using built-in machine learning models and algorithms to analyze and identify the data to be labeled assigned to it. During this process, the execution node will automatically identify key information in the data and match it with the corresponding label classification based on predefined labeling rules, such as keyword matching, classification algorithms, or clustering analysis. For example, if the task is to label corporate data, the execution node will automatically label the data into categories such as "small business", "medium-sized enterprise" or "large enterprise" based on the company's operating income, asset size, and other information.

[0062] After completing the labeling, the execution node updates the labeling results to the database and feeds back the task completion status and any necessary metadata to the task management module 103 to ensure the accuracy and traceability of the entire labeling process.

[0063] In this way, the task execution node 105 realizes automatic labeling of data, greatly improves processing efficiency and accuracy, and reduces the need for human intervention.

[0064] This embodiment realizes the rapid, accurate and automatic processing of a large amount of government data through the above-mentioned government big data automatic labeling system, significantly improves the data processing efficiency, reduces the labor cost, and enhances the consistency and accuracy of data labeling. First, this system realizes the effective acquisition of large-scale government data through the data source management module, providing a data basis for the labeling process. Secondly, the introduction of the label management module makes the creation and modification of label configuration information more flexible and accurate, and the system can dynamically adjust the label system according to data characteristics and business needs. The intelligent task creation and instruction generation of the task management module further improve the efficiency and accuracy of task allocation. The real-time monitoring and intelligent scheduling of the execution node management module ensure the balanced load and efficient execution of tasks among multiple service nodes. Finally, the task execution node labels the data according to the established rules to obtain the labeling results, realizing the automatic classification and labeling of the data. Overall, the system significantly improves the efficiency of government data labeling and reduces labor costs through automated technical means, while ensuring the accuracy and consistency of the labeling results, providing strong data support for government decision-making.

[0065] In some embodiments, the execution node management module 104 is specifically used to monitor the online status and load condition of the service node in real time, and determine the task execution node 105 according to the online status and the load condition.

[0066] In this embodiment, the execution node management module 104 tracks the online status and load status of each service node through a real-time monitoring mechanism. The execution node management module 104 can continuously collect data on node availability and workload using technologies such as heartbeat detection, resource utilization monitoring, and performance indicator analysis. For example, by monitoring key performance indicators such as CPU utilization, memory usage, network latency, and response time, the execution node management module 104 can accurately evaluate the current workload of each node.

[0067] In addition, by implementing fault detection and automatic recovery strategies, the execution node management module 104 can quickly respond to the offline or faulty state of the node to ensure the high availability of the system. When receiving a task instruction, the execution node management module 104 will comprehensively consider the online status and load conditions of all nodes, and use intelligent scheduling algorithms such as load balancing, priority allocation or failover to select the most suitable service node to execute the task. This dynamic task allocation strategy ensures that tasks can be executed efficiently and evenly among multiple nodes, while optimizing resource utilization and improving the response speed and stability of the entire labeling process.

[0068] In some embodiments, the service node is deployed in a distributed manner in multiple servers, and the execution node management module 104 is further used to provide registration services for the service node after the service node is deployed.

[0069] In this embodiment, the service node adopts a distributed deployment strategy in the government big data automatic labeling system, spanning multiple servers to enhance the scalability and fault tolerance of the system. The execution node management module 104 is not only responsible for real-time monitoring of the online status and load status of these service nodes, but also bears the important responsibility of service node life cycle management. When a new service node is deployed and ready to join the labeling work, the execution node management module 104 provides a registration service, allowing the node to register its own information with the system through a standardized registration process, including node identification, processing power, and current status. After successful registration, the service node will be included in the system's node pool, and will begin to receive task instructions from the execution node management module 104, and participate in the distributed processing of data labeling.

[0070] This registration mechanism ensures the flexibility and dynamic expansion capabilities of the system, while also facilitating the unified management and maintenance of service nodes.

[0071] In some embodiments, the task execution node 105 is further used to transmit the obtained annotation results to the result database 111 for storage after executing the annotation task.

[0072] Specifically, after successfully executing the labeling task, analyzing the data to be labeled and labeling the corresponding label classification according to the predefined labeling rules, the task execution node 105 is also responsible for transmitting these labeling results to the result database 111 for storage. This process involves storing the labeled data set or metadata, including label information and possible analysis results, in a structured form in the database 111 to ensure the persistence and traceability of the data. As a central repository, the result database 111 not only provides the system administrator with the ability to audit and monitor the labeling results, but also provides a basis for data sharing and further analysis for other modules or systems.

[0073] In this way, the task execution node 105 ensures the integrity and availability of the annotation results and supports the automation of government decision-making and business processes.

[0074] In some embodiments, the data source management module 101 is further used to associate and aggregate the data to be labeled according to data identification identifiers through a data exchange platform.

[0075] Specifically, the data source management module 101 is closely integrated with the data exchange platform (DXP) to use data identification identifiers, such as a unified social credit code, an organization code, or a project unique identifier, to identify and associate data to be labeled from different sources.

[0076] The data source management module 101 first extracts data from multiple scattered data sources, and then matches and associates these data according to predefined data models and identification identifiers through the interfaces and services of the data exchange platform. For example, if the data source includes tax records, enterprise registration information, and market supervision reports, the data source management module 101 will identify the enterprise identifiers in these data sets, and associate different data records of the same enterprise to form a comprehensive data view. In this way, the module can aggregate originally isolated data into a unified data set, providing accurate and consistent raw materials for subsequent automatic labeling and analysis. This process not only improves the availability and quality of data, but also provides richer and more comprehensive data support for government decision-making.

[0077] In some embodiments, the data to be annotated includes structured data and unstructured data.

[0078] For structured data, the data source management module 101 achieves efficient association and aggregation by defining a well-defined data model and query logic. These data are usually stored in a relational database with a fixed format and schema. The data source management module 101 uses SQL queries or specific data access APIs to identify and extract relevant data records according to preset data association rules, such as primary key and foreign key relationships. Among them, primary keys and foreign keys are key concepts in the database for ensuring data integrity and establishing relationships between tables. The primary key is a field or combination of fields in a table that uniquely identifies each record, while a foreign key is a field or combination of fields in a table that references the primary key of another table, thereby establishing a relationship between the two tables. The use of foreign keys not only ensures the referential integrity of the data, but also allows the database to automatically maintain the consistency of related data in the child table when updating or deleting records in the parent table.

[0079] For example, by using the unified social credit code as a correlation field, the data source management module 101 can retrieve and merge the multi-dimensional data of the same enterprise from the enterprise registration information, tax records, and market supervision database to form a comprehensive data set. In addition, the data source management module 101 may also apply data cleaning and conversion technology to ensure the consistency and accuracy of structured data from different data sources during the aggregation process.

[0080] The association and aggregation of unstructured data is more complicated because such data, such as text files, images, videos, etc., do not have a fixed format and structure. In this case, the data source management module 101 will use natural language processing (NLP), computer vision and other artificial intelligence technologies to identify and extract key information from the data. For example, for text data, the data source management module 101 can use text mining technology to identify and mark the subject, entity and sentiment tendency in the document; for image and video data, image recognition and scene analysis technology may be used to extract the features of visual content. Through these technologies, the data source management module 101 can abstract key information in unstructured data and associate it with structured data or other unstructured data, such as matching the events or scenes identified in the image with the text descriptions in news reports or social media posts to achieve cross-modal data aggregation.

[0081] The label verification module 106 is used to verify the accuracy of the labeling result according to the relationship between the labeling rules.

[0082] The mutual relationships may include but are not limited to logical relationships such as mutually exclusive relationships, subordinate relationships, and conditional dependent relationships, and may be one or more of them.

[0083] Verification of the accuracy of the annotation results can be obtained, which reflects the accuracy and reliability of the annotation results. The verification results are obtained by analyzing the mutual exclusive relationships, subordinate relationships, and conditional dependencies between the annotation rules, and using logical judgment, statistical analysis, or machine learning algorithms to determine whether the label of each data item correctly reflects its characteristics and classification. The verification results not only include confirmation of correct annotations, but also cover the identification of incorrect or suspicious annotations, providing a basis for the subsequent correction of annotation results, ensuring the quality of the data and the accuracy of the annotations.

[0084] In this embodiment, the label verification module 106 can retrieve the labeling results from the result database 111 to verify whether the labeled labels in the labeling results are accurate; it can also directly obtain the labeled labels from the task execution node 105 for verification after the task execution node 105 completes the labeling task.

[0085] The label verification module 106 is a key component in the government big data automatic labeling system, which is responsible for ensuring the accuracy and consistency of labels. The module performs the verification process by analyzing and understanding the complex relationships between labeling rules, such as mutually exclusive relationships, subordinate relationships, and conditional dependencies.

[0086] Specifically, the label verification module 106 can verify the accuracy of the labeled labels by analyzing and comparing the predefined relationships between the labeled labels in the labeling results. These relationships may include mutually exclusive relationships, ensuring that the same data item is not mistakenly assigned two labels that cannot coexist; subordinate relationships, ensuring that the hierarchical structure of the labels is correct, for example, a more specific label is only valid when its broader category label exists; and conditional dependencies, where the assignment of certain labels may depend on the existence or value of other labels. Logical judgments and rule engines can be used to automatically detect and verify these relationships. When contradictions or non-compliance with the rules are found, the module will mark potential errors and prompt manual review or automatic adjustment of labels to ensure the logic and accuracy of data labeling. This rule-based verification mechanism is a key link in ensuring the quality of government data labeling, which helps to reduce errors and improve data reliability.

[0087] For example, when processing enterprise data, if there is a mutually exclusive relationship, the label verification module 106 will check whether the same enterprise is mistakenly labeled as both a "small enterprise" and a "large enterprise"; in the case of a subordinate relationship, the label verification module 106 will verify whether an enterprise is labeled as "export-oriented", and whether its industry is reasonably labeled as "manufacturing"; and in a conditional dependency relationship, the label verification module 106 may verify whether the enterprise is eligible for "government subsidies" based on its "credit rating" label.

[0088] Through this intelligent verification mechanism, the label verification module 106 can automatically discover and correct errors in the labeling process, thereby improving the labeling quality and reliability of the entire system.

[0089] Combined with the above functional modules, the automatic labeling system for government big data provided by this embodiment realizes the automation and intelligence of data processing through the integration of multiple modules, and significantly improves the efficiency and accuracy of government data labeling. The data source management module ensures the efficient acquisition of data, the flexibility of the label management module allows rapid adaptation to new data features and business needs, and the task management module optimizes resource allocation through intelligent task creation and scheduling. The dynamic task allocation mechanism of the execution node management module ensures the rapid response and processing of tasks, and the precise execution of task execution nodes directly produces preliminary labeling results. Finally, the label verification module strictly verifies the labeling results by deeply analyzing the complex relationship between the labeling rules, ensuring high-quality output of data labeling. The present invention can automatically label a large amount of government data and verify the labeling results, which improves the efficiency of data labeling processing, reduces manual labeling operations, reduces labor costs, and enhances the consistency and accuracy of data labeling.

[0090] In some embodiments, the tag verification module 106 may automatically adjust the sensitivity of the verification rules according to the verification results through a logistic regression model.

[0091] The label verification module 106 is used to dynamically adjust the sensitivity of the verification strategy through the application of the logistic regression model to respond to the changing data characteristics and labeling requirements. This adaptive adjustment mechanism enables the system to optimize the model parameters according to the actual verification results, so as to more accurately identify and correct labeling errors. As time goes by and data accumulates, the system improves the accuracy of verification through continuous learning and optimization, reduces false positives and false negatives, and ensures the high accuracy and reliability of the government big data labeling results.

[0092] Among them, the sensitivity of the verification rules is adjusted to ensure that the government big data automatic labeling system can maintain better labeling quality and efficiency in different data sets and application scenarios. Due to the diversity and complexity of data, a single verification rule may not be able to adapt to all situations, resulting in too many false positives or false negatives in the verification results. By adjusting the sensitivity, correct labels and errors can be identified more accurately, thereby reducing the workload of manual review and improving the accuracy of automated labeling. In addition, as time goes by and data accumulates, data characteristics and labeling rules may change. Regularly adjusting the sensitivity of the verification rules helps the system adapt to these changes and ensure long-term labeling performance and data quality.

[0093] Specifically, the label verification module 106 can identify labeling errors that frequently occur during the data labeling process from the verification results by continuously monitoring and analyzing the verification results. The labeling errors include but are not limited to mislabeling, missing labels, or inconsistent labels.

[0094] Then, the label verification module 106 adjusts the sensitivity of the rules or retrains the verification model according to these labeling errors to better adapt to data changes or user labeling habits.

[0095] For example, if a particular tag error occurs frequently, the sensitivity of the verification rule to the tag labeling result can be enhanced. For example, the tag verification module 106 can increase the verification strictness of the tag, add more checkpoints or adjust the verification logic.

[0096] In this embodiment, the logistic regression model is a statistical model applied to binary classification problems, which can be used to estimate the probability of an event occurring.

[0097] Specifically, the logistic regression model may include the following formula:

[0098] ;

[0099] in:

[0100] p(y=1|x) is the probability that a sample belongs to the positive class (label is 1, i.e. y=1) given feature x. y represents the target variable, i.e. the verification result. In a binary classification problem, y can take two values, usually 0 and 1: y=1 represents the positive class, which means the predicted event occurs; y=0 represents the negative class, which means the predicted event does not occur.

[0101] Specifically, x is the model input feature, which can be obtained from the original data through data processing steps such as data cleaning, feature selection, feature extraction and feature scaling.

[0102] Among them, data cleaning includes removing noise, processing missing values, deduplication, and modification of the original data to ensure the quality of the data.

[0103] Feature selection is the process of selecting data features related to the target variable from the original data based on domain knowledge, statistical analysis or feature selection algorithms (such as recursive feature elimination, model-based feature selection, etc.).

[0104] Feature extraction is the transformation and encoding of selected features to make them suitable for model input. Specifically, text data is encoded, and text categories are converted into digital representations. It involves converting categorical data in text form into a numerical form that can be processed by machine learning models. For example, when processing enterprise data, the industry to which the enterprise belongs may be categorical data in text form, such as "manufacturing", "financial services", etc. In order to use these text categories in a logistic regression model, encoding techniques such as One-Hot Encoding can be used to convert each unique text category into a numerical vector. Suppose you want to process the text category feature of "industry type", which contains three different industry categories: "technology", "finance", and "manufacturing". Three new binary feature columns can be created, corresponding to these three industries. For a company that belongs to the "technology" industry, the "technology" column is set to 1, while the "finance" and "manufacturing" columns are both set to 0. Correspondingly, if the company belongs to the "finance" industry, the "finance" column is 1, and the rest are 0. In this way, each company will be assigned a unique numerical vector based on its industry type, for example, the vector corresponding to "technology" companies is [1, 0, 0], "finance" companies is [0, 1, 0], and "manufacturing" companies is [0, 0, 1]. This converts the text categories into numerical representations.

[0105] Feature scaling is to scale features so that they are in the same numerical range to avoid the model being overly sensitive to certain features. Common methods include minimum-maximum normalization or standardization.

[0106] After the above series of data processing, the related data can be combined into a vector x, which is a vector containing multi-dimensional data elements x n The vector can be expressed as x=[x1, x2, …,x n ].

[0107] Furthermore, y represents whether the data item is correctly labeled with a specific label. For example, if you are verifying whether a data item is correctly labeled as a "large enterprise", then: if the data item is indeed a "large enterprise", y will be labeled as 1. If the data item is not a "large enterprise", y will be labeled as 0. The logistic regression model predicts p(y=1|x) based on the input feature x, that is, the probability that the data item is correctly labeled as a positive class (for example, "large enterprise") given the feature x. This probability can then be used for decision making, such as setting a threshold. When the predicted probability exceeds this threshold, the prediction is considered to be a positive class, otherwise it is a negative class.

[0108] α is the intercept term of the model, which is equivalent to the output when all features x are 0.

[0109] β is the coefficient of the model, which indicates the influence of feature x on the output. Specifically, β can be composed of multiple model coefficients, expressed as β=[β1,.., β n ] T , n≥1,β1,..,β n are the elements x1,.., x in x respectively. n One-to-one corresponding model coefficients. Among them, α and β are model parameters, which can be obtained through the model training process, which involves using historical labeled data to optimize model parameters to minimize prediction errors. Specifically, at the beginning of training, α and β can be assigned an initial value respectively, and then the values ​​of α and β are continuously adjusted according to the gradient of the loss function by iteratively applying gradient descent or other optimization algorithms. Each iteration aims to improve the model parameters to more accurately predict the target variable y until the model performance reaches the optimal level or the stopping condition is met. The final determined α and β enable the model to predict the most accurate label classification probability p(y=1|x) based on the input feature x, thereby effectively supporting the label verification module to automatically adjust the sensitivity of the verification rules.

[0110] α+βx is a linear combination used for linear prediction, which converts the input feature x into a real value.

[0111] e is a natural constant, approximately equal to 2.71828.

[0112] Is a Sigmoid function, which is used to map the value of α+βx to the interval (0,1) to represent probability.

[0113] In the label verification module 106, if it is found that certain types of data are often mislabeled, the strictness of the verification of these data can be improved by adjusting the parameters (α and β) in the logistic regression model. Specifically, the weight of the mislabeled features can be increased (increasing the absolute value of β) to make the model more sensitive to these features, thereby improving the accuracy of the verification.

[0114] In this embodiment, the application of the logistic regression model provides a powerful prediction tool for the label verification module of the automatic labeling system for government big data, so that the system can accurately predict the probability p(y=1|x) that the verification result y is positive (i.e., the labeling is correct) based on the given feature x. By accurately estimating this probability, the model helps the system identify high-risk erroneous labels, thereby giving priority to manual review or automatic adjustment of these labels. In addition, the introduction of model parameters α and β enables the system to adjust the verification strategy according to data characteristics and optimize the sensitivity of the verification rules to adapt to different data characteristics and business needs. This data-driven approach not only improves the accuracy of the verification, but also enhances the system's adaptability, ensuring the high quality and efficiency of government data labeling.

[0115] The following example illustrates how to use a logistic regression model to further verify the accuracy of these annotation results and adjust the sensitivity of the verification strategy based on the verification results.

[0116] Assume that the automatic labeling system for government big data of this embodiment is used to label enterprise data, especially to identify enterprises in the science and technology industry. This embodiment can verify whether an enterprise belongs to the science and technology industry based on various characteristics of the enterprise (such as R&D investment, number of patents, product type, etc.), including the following steps:

[0117] (1) Initial model setting:

[0118] Feature vector x: x=[x1,x2,x3], which means that the input feature x includes the company’s R&D investment x1, number of patents x2, number of technical personnel x3, etc.

[0119] Determine the logistic regression model: including the following formula,

[0120] ;

[0121] Among them, y=1 means that the enterprise belongs to the technology industry, α is the intercept term, and β1, β2 and β3 are the model coefficients respectively.

[0122] (2) Verification process:

[0123] Data collection and labeling: Collect a batch of enterprise data and use the automatic labeling system to perform preliminary labeling on it.

[0124] Verification execution: Use the logistic regression model to verify these annotation results. The output of the model p(y=1|x) represents the probability that the company belongs to the technology industry.

[0125] Analyze verification results: Analyze the consistency between the probability output by the model and the actual annotations, and identify cases with incorrect annotations or high uncertainty.

[0126] (3) Adjust the verification strategy:

[0127] Identification issues: If it is found that the prediction accuracy of certain feature combinations is not high, or some businesses are incorrectly labeled or missed.

[0128] Adjustment coefficient: According to the verification results, adjust the values ​​of model parameters α, β1, β2 and β3. For example, if it is found that the number of patents is particularly important for distinguishing technology industry companies, the value of β2 can be increased to increase its weight in the model, thereby adjusting the model's sensitivity to the feature of the number of patents.

[0129] Retrain model: Retrain the model using the adjusted coefficients in the hope of improving the model's predictive accuracy on new data.

[0130] (4) Apply the adjusted model:

[0131] Re-verification: Use the adjusted model to annotate and verify new enterprise data to verify whether the adjustment is effective.

[0132] Continuous optimization: Based on the continuously collected verification results, the model coefficients are continuously adjusted and optimized to adapt to data changes and improve the accuracy of annotation.

[0133] Through the above process, the logistic regression model can verify the accuracy of these annotation results and adjust the sensitivity of the verification strategy according to the verification results to ensure the accuracy of the annotation results and the adaptive ability of the system. This data-driven approach improves the performance and reliability of the automatic annotation system for government big data.

[0134] In some embodiments, the tag verification module 106 may also have an adaptive learning mechanism, which can obtain the verification result after verifying the accuracy of the annotation result, and automatically adjust the verification strategy and modify the parameters in the verification strategy according to the verification result or user feedback (the user manually verifies the annotation result). Such an adaptive learning mechanism is usually based on a machine learning model, especially those models with online learning or incremental learning capabilities.

[0135] In some embodiments, the automatic adjustment function of the tag verification module 106 can be based on an adaptive learning mechanism and implemented by integrating adaptive learning algorithms that can dynamically optimize strategies and parameters based on verification results and user feedback. The following is a detailed process for implementing this function:

[0136] 1. Data collection: First, collect the verification result data, including error detection, user correction behavior, and annotation accuracy statistics.

[0137] 2. Error analysis: Analyze the collected data to identify common error types, false positives and false negatives, and the situations in which users are more likely to make corrections.

[0138] 3. Model training: Use machine learning techniques, such as supervised learning or reinforcement learning, to retrain the verification model based on the above analysis results to improve its ability to identify errors.

[0139] 4. Parameter optimization: Automatically adjust the parameters in the validation strategy, such as thresholds, weights, or rule sets, to better adapt to data characteristics and user behavior.

[0140] 5. User feedback loop: Establish a feedback mechanism to incorporate users’ corrective actions as part of the training data so that the system can learn and adapt to users’ expectations and standards.

[0141] 6. Continuous monitoring: After adjusting strategies and parameters, continuously monitor the verification results to ensure the effectiveness of improvement measures and provide a basis for further optimization.

[0142] 7. Automated deployment: Once the effectiveness of the adjusted validation strategies and parameters has been verified, these changes can be automatically deployed without manual intervention.

[0143] 8. Transparency and explanation: Ensure transparency of the adjustment process, provide explanations and rationales for the adjustments so that users understand why specific changes were made.

[0144] 9. Performance evaluation: Regularly evaluate the performance of the verification module, including indicators such as accuracy, recall rate, and F1 score, to ensure continuous improvement in verification quality.

[0145] Through this adaptive learning mechanism, the label verification module 106 can continuously evolve, improve its intelligence and efficiency, reduce reliance on manual review, and ensure the accuracy and consistency of data annotation.

[0146] In some embodiments, the label verification module 106 may also have an intelligent rule discovery function. By applying an unsupervised learning algorithm, the label verification module 106 can automatically identify and learn patterns and association rules in the data. The label verification module 106 analyzes correctly labeled data sets to identify common labeling patterns, abnormal patterns, and potential data distributions. For example, clustering algorithms are used to discover natural groupings in the data, or association rules are applied to learn to find potential relationships between labels. These discoveries can help the system propose new labeling rules or optimize existing rules, and even automatically identify labeling errors in the data without explicit guidance. In addition, intelligent rule discovery can also help the system adapt to new data characteristics or changes, and the verification logic can be updated without human intervention.

[0147] In some embodiments, the label verification module 106 may also have a multi-dimensional verification function, which can perform a more comprehensive verification process on the label accuracy. It not only considers the logical consistency between labels, but also includes the rationality of the time series, the accuracy of the spatial distribution and the semantic coherence.

[0148] For example, the label verification module 106 can check whether the labels of the same entity at different time points show a reasonable evolution trend, or whether the geographic location label is consistent with the actual geographic data. In addition, using natural language processing technology, the label verification module 106 can evaluate the semantic consistency of text data to ensure that the label can accurately reflect the deep meaning of the data.

[0149] Multi-dimensional verification can significantly improve the overall quality and credibility of data annotation.

[0150] In some embodiments, the label verification module 106 also has an automated repair suggestion function. When the label verification module 106 identifies labeling errors or inconsistencies, it not only marks the problem, but also provides possible repair solutions. This can be achieved by applying advanced data analysis techniques, such as decision trees, probabilistic models, or deep learning, to predict the most likely correct label. Repair suggestions can be generated based on contextual information, historical verification results, and user correction behavior. For example, if a company's "industry type" label is inconsistent with its "business scope" label, the label verification module 106 can suggest updating the "industry type" label to a value that matches the "business scope." This function can greatly improve the efficiency of the labeling process and reduce the workload of manual review and correction.

[0151] Among the related technologies, government big data has the special characteristics of high sensitivity, strict compliance requirements, extensive data relevance, and high requirements for data accuracy and real-time performance. These data usually contain personal privacy information, corporate operating data, and government decision-making basis, so special attention is paid to the security, accuracy, and legal use of data.

[0152] In view of these particularities of government big data, the label verification module 106 of this embodiment can provide strong safeguards for the labeling of these government big data through functions such as adaptive learning, intelligent rule discovery, multi-dimensional verification or automated repair suggestions. The label verification module 106 can ensure the accuracy and consistency of government data during the automatic labeling process, reduce the risk of decision-making errors due to incorrect labeling, and at the same time improve the efficiency and response speed of data processing, and meet the government decision-making requirements for data timeliness. In addition, the intelligent characteristics of the label verification module 106 also help to adapt to the dynamic changes of government data, providing the government with a flexible, reliable and efficient data labeling and verification tool, thereby enhancing the quality of government data management and the data support capabilities of government decision-making.

[0153] In some embodiments, Figure 2 As shown, the government big data automatic labeling system may also include one or more of a labeling task execution record module 107 , an information display module 108 , a report generation module 109 and a user management module 110 .

[0154] In some embodiments, the task execution recording module 107 is used to record the task execution status of the labeling task of the task execution node 105, and the information display module 108 is used to receive the task execution status and display the task execution status to the user.

[0155] Among them, the task execution record module 107 in the automatic labeling system acts as the auditor of the system. It records the execution of each labeling task in detail, including key information such as the start time and end time of the task, the amount of data processed, the accuracy of the labeling, the errors and exceptions encountered, etc. These records not only help the system administrator monitor and evaluate the progress and effect of the labeling tasks, but are also crucial for the continuous improvement and troubleshooting of the system.

[0156] The information display module 108 is a part of the user interface. It receives the task execution data provided by the task execution record module 107 and displays it to the user intuitively in the form of charts, reports or logs. In this way, the user can understand the status of the task, the quality of the annotation and the performance of the system in real time, so as to make timely decisions and adjustments. The design of the information display module 108 focuses on the user experience, ensuring that the information is presented clearly and is easy to understand, so that the user can quickly grasp the overall situation and details of the annotation task.

[0157] Through the collaborative work of these two modules, the automatic annotation system not only improves the efficiency and accuracy of data processing, but also enhances the transparency of the system and the friendliness of user interaction.

[0158] In some embodiments, the report generation module 109 is used to generate a government big data annotation report based on the task execution status and annotation results through a template engine and natural language technology.

[0159] Among them, the report generation module 109 is an intelligent component in the automatic annotation system of government big data. It automatically generates annotation reports by combining template engines and natural language generation technology.

[0160] The report generation module 109 first receives detailed task execution and annotation result data from the task execution record module 107, and then extracts key information and statistical data according to the predefined report template or the user-specified format. Using natural language processing technology, the module converts this data into a fluent and accurate text description while ensuring the professionalism and readability of the report content.

[0161] For example, the report generation module 109 may automatically summarize the overall accuracy of the annotations, data coverage, major trends or potential issues found, etc. In addition, the report generation module 109 can also add charts, images or other visual elements as needed to enhance the expressiveness and persuasiveness of the report.

[0162] Finally, the report output by the report generation module 109 not only comprehensively reflects the annotation status of the government big data, but also presents it in a format that is easy to understand and share, which greatly improves the efficiency and quality of report generation and reduces the workload of manual report writing.

[0163] In some embodiments, the user management module 110 is used to provide user management services and provide corresponding system access restrictions according to the user's authority. The user management service includes user authentication, role management and authority control.

[0164] The user management module 110 may limit the user's operation and editing authority on functional modules such as the tag management module 102 , the task management module 103 , and the execution node management module 104 .

[0165] For example, ordinary users may only be able to view the tag configuration information in the tag management module 102, but not be allowed to modify it; the task management module 103 may only allow authorized users to create and schedule tagging tasks; the operation of the execution node management module 104 may be limited to system administrators to monitor and maintain service nodes. Through this hierarchical authority management, the user management module 110 ensures the security of government data, avoids unauthorized access and potential data leakage risks, and ensures the transparency and accountability of system operations.

[0166] The user management module 110 can provide comprehensive user management services to ensure the security and compliance of the system by authenticating users, assigning roles and setting permissions. This module verifies the identity of users by implementing a user authentication mechanism to ensure that only authorized users can access the system. In addition, it is also responsible for role management, allowing system administrators to assign users to different roles based on their job responsibilities and needs, each with different responsibilities and access levels. The permission control function further refines the access restrictions to system resources to ensure that users can only access data and functions corresponding to their roles. For example, ordinary users may only be able to view reports and perform basic queries, while administrators can modify user management, system configuration, and data annotation rules.

[0167] Through these measures, the user management module 110 ensures the transparency and accountability of system operations while preventing unauthorized access and potential data leakage risks.

[0168] The automatic annotation system for government data in this embodiment is an integrated and automated solution that optimizes and simplifies the annotation process of government data through a series of collaborative modules.

[0169] The system starts with the data source management module 101, which is responsible for efficiently acquiring the data to be labeled from structured and unstructured data sources. Next, the label management module 102 automatically creates or updates the label configuration information, including label classification and labeling rules, according to the data characteristics and business needs. The task management module 103 further intelligently creates labeling tasks based on the data to be labeled and the label configuration information, generates task instructions, and is assigned to the most suitable service node for execution by the execution node management module 104. After receiving the task instructions, the task execution node 105 automatically labels the data using advanced machine learning technology and transmits the results to the result database for storage.

[0170] In addition, the system also includes a user management module 110 to ensure security and compliance, a task execution record module 107 to record task execution, and an information display module 108 to display task execution to users. The report generation module 109 automatically generates reports based on task execution and annotation results, while the predictive analysis module provides forward-looking data trend analysis.

[0171] Overall, this system has significantly improved the efficiency, accuracy and security of government data labeling through automation and intelligent technologies, while reducing labor costs and enhancing the flexibility and maintainability of the system.

[0172] The automatic annotation system for government big data in this embodiment has a wide range of application scenarios, as follows:

[0173] 1. Enterprise activity analysis:

[0174] Purpose: To strengthen the “precise” service and supervision of market players. Through technical means, “precisely” obtain the enterprise activity analysis model of different industries, regions and enterprise sizes.

[0175] Function: Help analyze the structural problems in regional economic development, accurately identify zombie enterprises for local governments, provide decision-making references for regional local governments, and then formulate corresponding policies to help them transform or exit the market.

[0176] 2. Risk warning analysis:

[0177] Purpose: The system monitors the credit status of enterprises or individuals and automatically identifies potential risk signals, such as legal proceedings, asset auctions, tax violations, etc.

[0178] Function: Provide risk warnings to financial institutions and regulatory authorities, helping them take early measures to reduce potential losses.

[0179] 3. Analysis of key investment projects:

[0180] Purpose: The system marks the relevant data of investment projects, including project progress, fund usage, expected benefits, etc.

[0181] Function: Enable government departments to effectively monitor key investment projects, ensure maximum investment benefits, and identify and resolve problems in a timely manner.

[0182] 4. Smart Approval:

[0183] Purpose: The system automatically marks various types of data in the approval process, such as the completeness and compliance of application materials.

[0184] Function: Improve the intelligence level of the approval process, reduce the workload of manual review, speed up the approval process, and improve government service efficiency.

[0185] 5. Policies benefiting enterprises:

[0186] Purpose: The system marks the government’s policies that benefit enterprises, and analyzes the qualifications and conditions of enterprises to achieve accurate matching of policies and enterprises.

[0187] Function: Help enterprises quickly understand and apply for applicable policy support, while enabling the government to more effectively implement measures to benefit enterprises and promote enterprise development and economic growth.

[0188] In each scenario, the government big data automatic data annotation system can provide efficient and accurate analysis results through automated data processing and intelligent annotation, thereby providing strong data support for government decision-making. In this way, the system not only improves the efficiency of government data processing, but also enhances the scientificity and accuracy of decision-making, which is of great significance to promoting the digital transformation of government affairs.

[0189] The present invention provides a method for automatically annotating government big data, which is applied to the automatic annotating system for government big data provided in any of the above embodiments, such as Figure 3 As shown, the method comprises the following steps:

[0190] Step 201, obtaining data to be labeled from government big data.

[0191] Specifically, the data source management module of the government big data automatic annotation system is responsible for extracting the required data to be annotated from the government data warehouse. These data may include structured data such as database records, and unstructured data such as text documents and multimedia files. The module uses efficient data extraction technology to ensure the integrity and consistency of the data and provide high-quality original materials for subsequent annotation work.

[0192] Step 202: Create or modify label configuration information according to the data to be labeled.

[0193] The tag configuration information includes tag classification and marking rules.

[0194] Specifically, the label management module of the government big data automatic labeling system allows users to create new label classifications or modify existing label configurations based on the characteristics of the new data to be labeled. These configurations not only define the classification labels of the data, but also contain specific labeling rules, such as the logic for classifying data based on specific conditions. This step is critical because it provides the necessary guidance and standards for automated labeling.

[0195] Step 203: Create a labeling task according to the data to be labeled and the label configuration information, and generate a task instruction.

[0196] Specifically, the task management module of the government big data automatic labeling system intelligently creates labeling tasks based on the defined data to be labeled and label configuration information. This module designs the execution process of the task, clarifies the goals and requirements of the task, and generates detailed task instructions. These instructions will guide the subsequent execution nodes on how to accurately complete the labeling work.

[0197] Step 204: after receiving the task instruction, determine a task execution node from a plurality of service nodes.

[0198] Specifically, after receiving the task instruction, the execution node management module of the government big data automatic annotation system selects the most suitable node to perform the annotation task according to the status and capabilities of the service nodes in the current system. This module adopts an intelligent scheduling algorithm, taking into account the load balancing, processing speed and historical performance of the nodes to ensure that the task can be completed efficiently and accurately.

[0199] Step 205: The labeling task is executed by the task execution node to label the data to be labeled with corresponding label classifications according to the labeling rule to obtain a labeling result.

[0200] Specifically, the task execution node selected in the government big data automatic labeling system starts to perform the labeling task according to the received task instructions and labeling rules. The automatic labeling engine on the node analyzes the data to be labeled, applies machine learning models and algorithms, matches the data with the corresponding label classification, and completes the labeling. After completion, these labeled data are stored or updated in the database for further analysis and decision-making. This process not only improves the speed and accuracy of labeling, but also reduces manual intervention and ensures the consistency and repeatability of data processing.

[0201] Step 206: Verify the accuracy of the annotation result based on the relationship between the annotation rules.

[0202] The accuracy of the annotation results is verified by analyzing and comparing the predefined relationships between the annotated labels in the annotation results. These relationships are logical relationships, which may include mutually exclusive relationships, ensuring that the same data item is not mistakenly assigned two labels that cannot coexist; subordinate relationships, ensuring that the hierarchy of labels is correct, for example, a more specific label is only valid when its broader category label exists; and conditional dependencies, where the assignment of certain labels may depend on the existence or value of other labels.

[0203] The module uses logical judgment and rule engines to automatically detect and verify these relationships. When contradictions or violations of rules are found, the module will mark potential errors and prompt manual review or automatic adjustment of labels to ensure the logic and accuracy of data annotation. This rule-based verification mechanism is a key link in ensuring the quality of government data annotation, helping to reduce errors and improve data reliability.

[0204] The technical effects of the automatic government big data labeling method of this embodiment can refer to the technical effects of the automatic government big data labeling system of the above-mentioned embodiment, and will not be repeated here.

[0205] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A government big data automatic labeling system, characterized in that: include: The data source management module is used to obtain the data to be labeled from the government big data; A tag management module, used to create or modify tag configuration information according to the data to be tagged, wherein the tag configuration information includes tag classification and tagging rules; A task management module, used to create a labeling task according to the data to be labeled and the label configuration information, and generate a task instruction; An execution node management module, used to determine a task execution node from a plurality of service nodes after receiving the task instruction; The task execution node is used to execute the labeling task to label the data to be labeled with corresponding label classifications according to the labeling rules to obtain a labeling result; The label verification module is used to verify the accuracy of the annotation results according to the mutual relationship between the annotation rules, and the mutual relationship includes one or more of mutually exclusive relationships, subordinate relationships and conditional dependencies; the label verification module also has an adaptive learning mechanism, which is used to obtain the verification results after verifying the accuracy of the annotation results, and automatically adjust the verification strategy according to the verification results or user feedback, and modify the parameters in the verification strategy.

2. The automatic annotation system for government big data according to claim 1 is characterized in that: The label verification module is also used to adjust the sensitivity of the verification strategy through a logistic regression model according to the verification result obtained by verifying the accuracy of the annotation result.

3. The automatic annotation system for government big data according to claim 2 is characterized in that: The logistic regression model may include the following formula: Among them, y is the target variable, representing the verification result; p(y=1|x) represents the probability that y=1 when the feature x is given; α is the intercept term of the model, and β is the coefficient of the model.

4. The automatic annotation system for government big data according to claim 1 is characterized in that: The execution node management module is specifically used to monitor the online status and load status of the service node in real time, and determine the task execution node according to the online status and the load status.

5. The automatic annotation system for government big data according to any one of claims 1 to 4, characterized in that: The service nodes are distributedly deployed in multiple servers, and the execution node management module is further used to provide registration services for the service nodes after the service nodes are deployed.

6. The automatic annotation system for government big data according to claim 1 is characterized in that: It also includes a task execution recording module and an information display module. The task execution recording module is used to record the task execution status of the labeling task, and the information display module is used to receive the task execution status and display the task execution status to the user.

7. The automatic annotation system for government big data according to claim 1 is characterized in that: The task execution node is also used to transmit the annotation result to the result database for storage after executing the annotation task.

8. The automatic marking system for government big data according to claim 6 or 7 is characterized in that: It also includes a report generation module, which is used to generate a government big data annotation report based on the task execution status and the annotation results.

9. The automatic annotation system for government big data according to claim 1 is characterized in that: The data source management module is also used to associate and aggregate the data to be labeled according to data identification identifiers through a data exchange platform.

10. A method for automatic annotation of government big data, characterized in that: Applied to the government big data automatic annotation system according to any one of claims 1 to 9, the method comprising: Obtain data to be labeled from government big data; Creating or modifying label configuration information according to the data to be labeled, wherein the label configuration information includes label classification and labeling rules; Creating a labeling task according to the data to be labeled and the label configuration information, and generating a task instruction; After receiving the task instruction, determining a task execution node from a plurality of service nodes; Executing the labeling task through the task execution node to label the data to be labeled with corresponding label classifications according to the labeling rules to obtain a labeling result; The accuracy of the annotation results is verified according to the mutual relationship between the annotation rules.

Images