Government affair data acquisition method, device, equipment and system

Abstract

The invention provides a government affair data collection method, device, equipment and system, and is applied to a data collection system. The method comprises the following steps: obtaining a to-be-executed task; monitoring the monitoring objects influencing the execution of the to-be-executed task according to the monitoring rule, and generating monitoring data information of each monitoring object; for the monitoring data information of each monitoring object, according to a comparison parameter rule, the monitoring data information is analyzed to obtain a data analysis result, and the comparison parameter rule is a regular expression and a monitoring parameter threshold corresponding to the monitoring object; determining an optimization adjustment scheme of the monitoring object according to the data analysis result; and performing optimization adjustment on the monitoring object according to the optimization adjustment scheme, and after the optimization adjustment of the data acquisition system is completed, executing the to-be-executed task to complete data acquisition. The method is used for improving the success rate and the data quality of data acquisition, enhancing the adaptability of the system to different acquisition requirements and environment changes, and improving the data acquisition efficiency.

Description

Technical Field

[0001] The present application relates to the field of data collection technology, and in particular to a method, device, equipment and system for collecting government data. Background Art

[0002] With the development of cloud computing, big data and artificial intelligence, the collection and processing of various data in cross-regional, cross-departmental and cross-industry government information systems have also entered the era of big data. As the volume of government data continues to expand and the data is changing rapidly, the requirements for the collection efficiency and success rate of data collection systems are becoming higher and higher.

[0003] The existing government data collection methods mainly include configuring data collection tasks and configuring scheduling tasks. According to the scheduling task trigger, a one-time or periodic data collection execution task is performed. If there is an abnormality in the data collection result, the operation and maintenance personnel will analyze it according to the task monitoring log, and handle the exception according to the analysis result. The exception handling is carried out according to different abnormal tasks. After the data collection exception handling is completed, the data collection task will be re-performed.

[0004] However, the existing government data collection methods have the problem of low data collection success rate when performing data collection. Summary of the invention

[0005] The present application provides a government data collection method, device, equipment and system to solve the problem of low data collection success rate in existing government data collection methods during data collection, so as to improve the performance and reliability of the data collection system.

[0006] In a first aspect, the present application provides a method for collecting government data, comprising:

[0007] Get the tasks to be executed;

[0008] According to the monitoring rules, the monitoring objects that affect the execution of the tasks to be executed are monitored, and monitoring data information of each monitoring object is generated. The monitoring objects include the equipment status, task status, metadata and manual exception handling records related to the tasks to be executed in the data acquisition system. The monitoring rules include the specific types and monitoring parameters of the monitoring objects;

[0009] For each monitored object, the monitored data information is analyzed according to the comparison parameter rule to obtain a data analysis result. The comparison parameter rule is a regular expression corresponding to the monitored object and a monitoring parameter threshold.

[0010] Determine the optimization adjustment plan for the monitored object based on the data analysis results;

[0011] According to the optimization and adjustment plan, the monitored object is optimized and adjusted, and after the optimization and adjustment of the data acquisition system is completed, the tasks to be executed are executed to obtain the collected data.

[0012] In a possible implementation, obtaining the task to be executed includes:

[0013] Responding to the configuration operation of the system personnel, obtaining the data collection task;

[0014] Determine the scheduling task type in the data collection task;

[0015] If the scheduling task type is a periodic task, determine the execution cycle of the data collection task and the tasks to be executed corresponding to the execution cycle;

[0016] If the scheduling task type is a one-time task, the data collection task is determined as a task to be executed.

[0017] In a possible implementation, the monitoring data information is analyzed according to the comparison parameter rule to obtain a data analysis result, including:

[0018] According to the comparison parameter rules, determine the regular expression and monitoring parameter threshold corresponding to the monitoring object;

[0019] Based on the regular expression, a matching analysis is performed on the monitoring data information of the monitoring object to determine a matching analysis result, where the matching analysis result is used to indicate whether the monitoring data information includes a preset monitoring parameter;

[0020] If the matching analysis result indicates that the monitoring information includes preset monitoring parameters, the preset monitoring parameters are compared and analyzed with the parameter thresholds respectively corresponding to the monitoring parameter thresholds to determine the comparison analysis result, which indicates whether the monitoring parameters of the monitored object exceed the monitoring parameter thresholds;

[0021] The data analysis results are generated according to the matching analysis results and the comparison analysis results.

[0022] In a possible implementation, after determining an optimization adjustment plan for the monitored object according to the data analysis result, the method further includes:

[0023] According to the optimization adjustment plan of each monitored object, determine the final optimization adjustment plan related to the task to be performed in the data acquisition system;

[0024] Accordingly, according to the optimization and adjustment plan, the monitored objects are optimized and adjusted, including:

[0025] According to the final optimization and adjustment plan, each monitored object is optimized and adjusted.

[0026] In a possible implementation, according to the optimization adjustment scheme of each monitored object, determining the final optimization adjustment scheme related to the task to be performed in the data acquisition system includes:

[0027] Extract features of the optimization adjustment scheme for each monitored object to obtain feature representation of each monitored object;

[0028] Based on the self-attention mechanism and the pre-trained large model, the feature weight corresponding to each monitored object is determined;

[0029] The final optimization adjustment plan is generated based on the feature representation and feature weight of each monitored object.

[0030] In a possible implementation, optimizing and adjusting each monitoring object according to the final optimization and adjustment plan includes:

[0031] If the final optimization adjustment plan indicates that manual adjustment is required, the final optimization adjustment plan is sent to the system personnel, and in response to the plan adjustment operation of the system personnel, a manual optimization adjustment plan is obtained, and each monitored object is optimized and adjusted according to the manual optimization adjustment plan;

[0032] If the final optimization adjustment plan indicates that no manual adjustment is required, each monitored object is automatically optimized and adjusted according to the final optimization adjustment plan.

[0033] In a possible implementation, executing the task to be executed to obtain the collected data includes:

[0034] Based on the tasks to be executed, data in the source database is collected to obtain initial collected data;

[0035] According to the data conversion rules in the task to be executed, the initial collected data is converted to obtain the collected data.

[0036] In a possible implementation, the method further includes:

[0037] Determine the execution results of the tasks to be executed based on the collected data;

[0038] If the execution result indicates that the task execution failed, the monitoring data information and optimization adjustment plan of the task to be executed will be sent to the system personnel, and the failure analysis results and manual processing plan of the system personnel will be received;

[0039] According to the manual processing plan, re-execute the task to be executed and obtain the collected data after execution;

[0040] The execution process information of the task to be executed is stored, and the execution process information includes monitoring data information, optimization adjustment plan, failure analysis results, and manual processing plan.

[0041] In a second aspect, the present application provides a government data collection device, comprising:

[0042] The acquisition module is used to obtain tasks to be executed;

[0043] The monitoring module is used to monitor the monitoring objects that affect the execution of the tasks to be executed and generate monitoring data information of each monitoring object. The monitoring objects include the equipment status, task status, metadata and manual exception handling records related to the tasks to be executed in the data acquisition system. The monitoring rules include the specific types of monitoring objects and monitoring parameters.

[0044] An analysis module is used to analyze the monitoring data information of each monitored object according to a comparison parameter rule to obtain a data analysis result. The comparison parameter rule is a regular expression corresponding to the monitored object and a monitoring parameter threshold;

[0045] A determination module is used to determine an optimization adjustment plan for the monitored object based on the data analysis results;

[0046] The processing module is used to optimize and adjust the monitored object according to the optimization and adjustment plan, and after the optimization and adjustment of the data acquisition system is completed, execute the tasks to be executed to obtain the collected data.

[0047] In a third aspect, the present application provides an electronic device, including: a memory, a processor;

[0048] Memory stores computer-executable instructions;

[0049] The processor executes the computer-executable instructions stored in the memory to implement the above first aspect and / or various possible implementations of the first aspect.

[0050] In a fourth aspect, the present application provides a data acquisition system, including a data acquisition prediction optimization device and a data acquisition device;

[0051] The data acquisition prediction and optimization device is used to monitor and optimize the monitored object;

[0052] The data acquisition device is used to execute the task to be executed and obtain the collected data.

[0053] In a possible implementation, the data acquisition prediction optimization device includes a monitoring module, an analysis module, and an optimization module;

[0054] The monitoring module is used to obtain tasks to be executed, and according to monitoring rules, monitor the monitoring objects that affect the execution of the tasks to be executed, and generate monitoring data information of each monitoring object. The monitoring objects include the equipment status, task status, metadata and manual exception handling records related to the tasks to be executed. The monitoring rules include the specific type of the monitoring object and the monitoring parameters.

[0055] The analysis module is used to analyze the monitoring data information of each monitored object according to the comparison parameter rule to obtain the data analysis result. The comparison parameter rule is the regular expression and monitoring parameter threshold corresponding to the monitored object, and determine the optimization adjustment plan of the monitored object according to the data analysis result;

[0056] The optimization module is used to optimize the monitored object according to the optimization adjustment plan.

[0057] In a fifth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed, they are used to implement the above first aspect and / or various possible implementations of the first aspect.

[0058] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed, implements the above first aspect and / or various possible implementations of the first aspect.

[0059] The present application provides a government data collection method, device, equipment and system, which obtains tasks to be executed; monitors the monitoring objects that affect the execution of the tasks to be executed according to the monitoring rules, and generates monitoring data information of each monitoring object, wherein the monitoring objects include the equipment status, task status, metadata status and manual exception handling records related to the tasks to be executed in the data collection system, and the monitoring rules include the specific type and monitoring parameters of the monitoring objects; for the monitoring data information of each monitoring object, analyze the monitoring data information according to the comparison parameter rules to obtain the data analysis results, and the comparison parameter rules are the regular expressions and monitoring parameter thresholds corresponding to the monitoring objects; according to the data analysis results, determine the optimization adjustment of the monitoring objects Plan; According to the optimization and adjustment plan, the monitored objects are optimized and adjusted, and after the optimization and adjustment of the data acquisition system is completed, the means of executing the tasks to be executed to obtain the collected data are used to effectively monitor the various factors affecting the execution of tasks in the data acquisition system, and then combined with detailed monitoring rules and real-time data analysis, regular expressions and thresholds are used for in-depth data comparison to identify possible abnormal situations in the system. Based on these analysis results, the system generates an optimization and adjustment plan and adjusts the monitored objects to improve performance and avoid the occurrence of abnormal situations, thereby improving the success rate and data quality of data collection, and enhancing the system's adaptability to different collection requirements and environmental changes, realizing an efficient automated data collection process. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0061] Figure 1 A schematic diagram of a data acquisition system provided for this application;

[0062] Figure 2 A flowchart of a government data collection method provided for this application;

[0063] Figure 3 A specific flow chart of a government data collection method provided for this application;

[0064] Figure 4 A schematic diagram of the structure of the government data collection device provided for this application;

[0065] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0066] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0068] In the prior art, the main modules of the government data collection system include a data collection module, a task scheduling module and a task monitoring module, among which the main functions of the data collection module include data reading, data conversion and data writing functions; the task scheduling module mainly realizes the periodic triggering of the collection tasks configured by the collection module according to the plan; the task monitoring module mainly monitors the scheduling tasks and collection tasks, and outputs detailed log information such as task success, failure and exception for analysis and use by the collection system users, and is used to analyze and troubleshoot the causes of task exceptions, and then handle the abnormal tasks.

[0069] Correspondingly, the method for collecting government data based on the existing government data collection system includes: the data collection system configures the data source information through the network and establishes a connection with the data source information; after the data collection system establishes a connection with the data source, the data is read from the data source to the data collection system through the network; then, the data read according to the rules configured in the data collection system is processed and converted to form result data; finally, the processed result data is written to the target database through the data source connection. After the data collection task is completed, it is necessary to determine whether there are abnormalities in the collected data; if there are abnormalities, the operation and maintenance / operation personnel need to analyze according to the task monitoring log and handle the exceptions based on the analysis results. Furthermore, the exceptions include equipment and network abnormalities, data collection system abnormalities, data abnormalities, etc., and the exceptions are handled in a targeted manner according to different abnormal tasks; after the exceptions are handled, the data collection task is re-performed. However, performing exception handling after government data collection will make the success rate of data collection uncontrollable, affected by equipment status, collection and task scheduling configuration, data anomalies, etc. At the same time, the existing exception handling mechanism mainly relies on manual post-analysis, that is, when the data collection task fails, the operation and maintenance / operation personnel will intervene to investigate the cause and take measures. This passive processing method is inefficient and may miss the best time for timely repairs. It also has high technical requirements for operation and maintenance / operation personnel. Therefore, the current data collection system faces the problem of high failure rate of data collection tasks when performing data collection.

[0070] In order to solve the above problems, the embodiments of the present application provide a government data collection method, device, equipment and system, which ensures detailed control of the entire data collection process and generates monitoring data information by comprehensively monitoring the monitoring objects such as the equipment status, task status, metadata status and manual exception handling records related to the tasks to be executed in the data collection system; by using comparison parameter rules such as regular expressions and monitoring parameter thresholds, the monitoring data information can be deeply analyzed to timely discover abnormal patterns in the data, thereby preventing potential failures; based on the data analysis results obtained by analysis, an optimization adjustment plan for the monitored objects is formulated and implemented, which helps to solve problems that may affect the execution of data collection tasks in advance, thereby improving the success rate of the tasks; and then through automated task scheduling, it is ensured that the data collection tasks are automatically executed according to the predetermined plan, reducing human errors and delays, and improving the efficiency and accuracy of data collection; monitoring manual exception handling records helps the system analyze and learn from past failure cases, thereby avoiding similar problems in future data collection tasks.

[0071] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0072] Figure 1 A schematic diagram of a data acquisition system provided for this application, such as Figure 1 As shown, the data acquisition system of the present application includes a data acquisition prediction optimization device and a data acquisition device. The data acquisition prediction optimization device is used to monitor and optimize the monitored object; the data acquisition device is used to execute the task to be executed to obtain the collected data.

[0073] Among them, the data acquisition prediction optimization device may include a monitoring module, an analysis module and an optimization module;

[0074] The monitoring module is used to obtain tasks to be executed, and according to monitoring rules, monitor the monitoring objects that affect the execution of the tasks to be executed, and generate monitoring data information of each monitoring object. The monitoring objects include the equipment status, task status, metadata and manual exception handling records related to the tasks to be executed. The monitoring rules include the specific type of the monitoring object and the monitoring parameters.

[0075] The analysis module is used to analyze the monitoring data information of each monitored object according to the comparison parameter rule to obtain the data analysis result. The comparison parameter rule is the regular expression and monitoring parameter threshold corresponding to the monitored object, and determine the optimization adjustment plan of the monitored object according to the data analysis result;

[0076] The optimization module is used to optimize the monitored object according to the optimization adjustment plan.

[0077] In one example, the monitoring module mainly monitors the device status, task status, metadata status and manual exception processing records, and records the current device (server, storage, network and other equipment) status, collection and scheduling task status, source database, target database, and abnormal manual processing records in real time, including the current situation and historical records, to provide data and basis for subsequent analysis modules, mainly including modules such as device status monitoring, task status monitoring, metadata monitoring, and manual processing records.

[0078] Analysis module: By analyzing the device status data, task status data, metadata, and manually processed data, we determine the situations that may lead to data collection failure or abnormality. Furthermore, we first analyze the device status data, task status data, metadata, and manually processed data individually, and combine the device parameter threshold (server load threshold, storage load threshold, network connection status, etc.) settings, task load conditions, whether the metadata has changed, and the experience and records of manual processing of task abnormalities to analyze and determine the situations that may lead to data collection failure or abnormality. Then, we comprehensively consider the situations that may lead to data collection failure or abnormality, and predict in advance the specific situations that may lead to the failure of the data collection task, so as to provide an analytical basis for the subsequent optimization module.

[0079] Optimization module: Based on the data provided by the analysis module, equipment optimization adjustment rules, task optimization adjustment rules, and metadata optimization adjustment rules are given according to the optimization rule model. Combined with the big model, comprehensive analysis is conducted to determine the best optimization adjustment plan. The optimization adjustment execution can choose to execute the optimization adjustment automatically or manually make a secondary judgment.

[0080] It should be noted that, in this system, the data acquisition device may include:

[0081] Data collection module: implements data collection task configuration, and executes data collection jobs according to task scheduling triggers, reads data from the data source, converts the data into result data according to the configured data conversion rules, and finally writes the result data to the target database.

[0082] Task scheduling module: It enables tasks to be started at scheduled times according to the plan. It can set the execution time of the collection tasks, whether there are dependencies between tasks, etc. It can also implement one-time execution and periodic execution of collection tasks, as well as retry of task failures and distributed scheduling strategy configuration.

[0083] Task monitoring module: monitors data collection tasks and task scheduling, monitors whether tasks are successfully executed as planned, issues abnormal reminders for abnormal tasks, and logs the entire process for analysis and processing by maintenance and operation personnel.

[0084] Through the collaborative work of various modules, the system can: monitor and analyze each link in the data collection process in real time, identify and predict potential problems; provide automated and intelligent optimization and adjustment solutions to ensure that the system runs in the optimal state; support flexible task scheduling and monitoring, and monitor, analyze and predict the optimization of equipment, tasks, metadata in advance before the data collection task is started, and perform optimization in advance to improve the success rate and efficiency of data collection.

[0085] Figure 2This is a flowchart of a method for collecting government data provided by this application. The execution subject of this method can be a server storing a data collection system or other servers, and this embodiment is not particularly limited here. Figure 2 As shown, the method may include:

[0086] S201, obtaining tasks to be executed.

[0087] In this step, the tasks to be executed may be specific data acquisition tasks in the data acquisition device. These tasks include the task execution information configured by the data acquisition module to extract data from the data source, process the data according to predefined conversion rules, and write the processed data into the target database. They also include the task scheduling module settings to manage and schedule the execution time of the task, such as timed start, dependency management, and failure retry mechanism.

[0088] When the specific data collection tasks are set up, the data collection prediction and optimization device obtains these tasks from the data collection device, thereby monitoring the monitoring objects related to the tasks, analyzing and optimizing these tasks in advance to ensure that they can be executed smoothly, thereby improving the success rate and efficiency of the data collection tasks.

[0089] Optionally, the data acquisition prediction and optimization device can be integrated with the task scheduling module to obtain a list of current tasks to be executed by accessing the task queue or task management system; the task information can also be stored in a database, and the prediction and optimization device can obtain the tasks to be executed by querying the database, for example, querying the records in the task table with a status of "to be executed"; it can also obtain detailed information on the tasks to be executed by calling the API interface provided by the task scheduling module; it can also monitor the event notification of the task scheduling module based on an event-driven architecture to obtain information on the tasks to be executed.

[0090] In this embodiment of the present application, the method for obtaining the task to be executed may include:

[0091] Responding to the configuration operation of the system personnel, obtaining the data collection task;

[0092] Determine the scheduling task type in the data collection task;

[0093] If the scheduling task type is a periodic task, determine the execution cycle of the data collection task and the tasks to be executed corresponding to the execution cycle;

[0094] If the scheduling task type is a one-time task, the data collection task is determined as a task to be executed.

[0095] The configuration operation of the system personnel may refer to the operation of the maintenance / operation personnel of the data acquisition system to set up or adjust the system through the user interface or configuration file, for example, configuring a new data acquisition task or modifying the parameters of an existing task.

[0096] A data collection task may refer to specific task execution information of extracting data from a data source, processing data, and storing it in a target database, and may include data source connections, data conversion rules, and data storage targets. Among them, a data source may include a source data source where data is initially collected and stored.

[0097] The scheduling task type can be the execution type of the task plan managed and scheduled by the task scheduling module, such as periodic tasks and one-time tasks. For example, a periodic task is repeatedly executed at a set time interval, such as once every day at 2 a.m.; a one-time task is only executed once, such as at a specific date and time.

[0098] The to-be-executed task corresponding to the execution cycle may be a periodic task that is about to reach the execution time among the tasks that have not yet been executed.

[0099] In one example, the system provides a user interface that allows system personnel to configure new data collection tasks. For example, the system personnel input task parameters (such as source data source, conversion rules, target database, etc.), and the system saves these configurations to the task management database; then, the system reads the task configuration from the task management database and checks the scheduling type field of the task to determine whether the task is a periodic task or a one-time task; the system calculates the next execution time based on the execution cycle field (for example, daily, weekly) in the task configuration, and marks it as a task to be executed. For example, if the current time is January 1 and the task execution cycle is daily, the system marks the task on January 2 as to be executed; at the same time, the system checks the planned execution time of the one-time task. If the time is after the current time and close to the current time, the task is marked as a task to be executed. For example, if the current time is January 1 and the planned execution time is January 2, the system marks the task as to be executed.

[0100] By distinguishing between periodic tasks and one-time tasks, the system can accurately determine the execution time of each task, ensuring that the task is executed within the expected time window, avoiding omission or duplication of tasks. At the same time, it can avoid resource conflicts and overloads, and facilitate the formulation of subsequent system optimization plans. For example, periodic tasks can be scheduled to be executed during time periods with low system load, thereby optimizing the use of server, storage and network resources.

[0101] S202. According to the monitoring rules, the monitoring objects that affect the execution of the tasks to be executed are monitored, and monitoring data information of each monitoring object is generated. The monitoring objects include the equipment status, task status, metadata and manual exception handling records related to the tasks to be executed in the data acquisition system. The monitoring rules include the specific type of the monitoring object and the monitoring parameters.

[0102] Among them, monitoring rules can be specific standards and parameters for monitoring various objects in the system. Monitoring rules define which objects need to be monitored, how to monitor, and which parameters need attention. Monitoring objects can refer to entities or elements that need to be monitored in the system. Monitoring data information can refer to data collected through the monitoring process, reflecting the current and historical status of the monitored objects. This information is used for analysis and decision-making.

[0103] For example, the specific types that need to be monitored in the device status may be servers, storage devices, network devices, etc., and the monitoring parameters may include CPU usage, memory usage, disk space, network traffic, network load, congestion, delay data, etc.; the specific types that need to be monitored in the task status may be specific collection task names, scheduling task names, and the monitoring parameters may include task execution time, task success rate, data transmission volume, task scheduling frequency, etc.; the specific types that need to be monitored in the metadata situation may be specific metadata types or names, and the monitoring parameters may include metadata update frequency, metadata consistency, etc.; the specific types that need to be monitored in the manual exception processing records may be specific manual processing record types or names, and the monitoring parameters may include processing time, processing results, etc.

[0104] Optionally, the monitoring rules can be continuously adjusted based on the actual system status, current task status, and historical monitoring data information of the system to adapt to new changes and ensure the stable execution of data collection tasks.

[0105] Real-time monitoring provides comprehensive visibility into the system's operating status, enabling system personnel to quickly understand the system's health status. The collected monitoring data can also provide a basis for analysis and optimization to enhance the system's stability and reliability and reduce the risk of mission failure.

[0106] S203. Analyze the monitoring data information of each monitored object according to the comparison parameter rule to obtain a data analysis result. The comparison parameter rule is a regular expression corresponding to the monitored object and a monitoring parameter threshold.

[0107] Among them, the comparison parameter rules can be the standards and methods used to analyze monitoring data information. Regular expressions can be used to match and extract data in a specific format or pattern. Monitoring parameter thresholds can be used to determine whether monitoring data exceeds the numerical limit of the normal range. For example, CPU usage that continues to exceed 80% may be considered abnormal.

[0108] By comparing parameter rules, the system can accurately identify abnormal situations that are beyond the normal range, and thus respond and handle them quickly, which can reduce reliance on manual monitoring and improve operation and maintenance efficiency and response speed.

[0109] Among them, in the embodiment of the present application, the method of analyzing the monitoring data information according to the comparison parameter rule to obtain the data analysis result may include:

[0110] According to the comparison parameter rules, determine the regular expression and monitoring parameter threshold corresponding to the monitoring object;

[0111] Based on the regular expression, a matching analysis is performed on the monitoring data information of the monitoring object to determine a matching analysis result, where the matching analysis result is used to indicate whether the monitoring data information includes a preset monitoring parameter;

[0112] If the matching analysis result indicates that the monitoring information includes preset monitoring parameters, the preset monitoring parameters are compared and analyzed with the parameter thresholds respectively corresponding to the monitoring parameter thresholds to determine the comparison analysis result, which indicates whether the monitoring parameters of the monitored object exceed the monitoring parameter thresholds;

[0113] The data analysis results are generated according to the matching analysis results and the comparison analysis results.

[0114] In this step, the monitoring parameter threshold value may be a numerical limit for judging whether the monitoring data exceeds the normal range. It should be noted that when judging whether the monitoring data exceeds the numerical limit of the normal range, the judgment may be made based on the monitoring data at the preset time, or it may be made based on the monitoring data at each time in the preset time period for continuous judgment. Wherein, when continuous judgment is made based on the monitoring data at each time in the preset time period, optionally, the alarm or optimization level is set according to the number of monitoring data exceeding the threshold value at each time in the preset time period, so that the more the number of monitoring data exceeding the threshold value at each time in the preset time period, the more urgent processing is required. For example, the threshold value of CPU usage may be set to 90%. If the CPU usage at the preset time in the matching analysis result is 85%, or the CPU usage at each time in the preset time period is 85%, it can be known through comparison analysis that the current CPU usage does not exceed the threshold, then the data analysis result may represent the situation where the CPU usage does not trigger an alarm or optimization. For another example, the CPU usage at each time in the preset time period is 91%. It can be known through comparison analysis that the current CPU usage continues to exceed the threshold, then the data analysis result may represent the situation where the CPU usage triggers an alarm or optimization.

[0115] In one example, a regular expression can match preset monitoring parameters in monitoring data information to determine whether the preset monitoring parameters are monitored and generate monitoring data. If monitoring data exists, the specific data of the preset monitoring parameters is compared and analyzed to see whether they exceed the corresponding parameter threshold. If monitoring data does not exist, it means that the preset monitoring object may have a fault or abnormality, and the matching analysis result is a matching failure, which provides a reference for the formulation of subsequent optimization and adjustment plans.

[0116] Through regular expressions and threshold comparisons, the system can accurately identify abnormal situations and ensure timely response and processing. The data analysis results provide a reliable basis for system optimization and decision-making.

[0117] S204: Determine an optimization adjustment plan for the monitored object based on the data analysis results.

[0118] Furthermore, based on the data analysis results, the system personnel or the automated system formulates an optimization plan, which may include specific adjustment measures and expected results. For example, if the analysis results show that the server load is too high, the optimization plan may include increasing server resources or adjusting the task scheduling time. The automated system formulates the plan by analyzing the optimization logic in the historical processing plan and generating the optimization adjustment rules corresponding to the monitored object. As the historical processing plan increases over time, the optimization adjustment rules can be automatically optimized and iterated as the system is used, thereby improving the feasibility and accuracy of the automated system's plan.

[0119] Optionally, based on the optimization and adjustment rules, according to the data analysis results and historical monitoring data information, the status of the monitored object at the current moment and the development trend of the status over time are analyzed to predict whether the status of the corresponding monitored object at the next moment or the next time period (including the time period when the task to be executed is about to be executed) is in a normal state, so as to formulate an optimization and adjustment plan that can ensure the successful execution of the task.

[0120] For example, the optimization and adjustment plan for device status may involve adjusting the server's CPU, memory, storage capacity and other configurations to meet the system's performance requirements, or adjusting the network bandwidth and other network parameters to ensure the speed and stability of data transmission; the optimization and adjustment plan for task status may involve adjusting the number of concurrent data collection tasks to optimize resource utilization and task execution efficiency, or staggering the execution time of scheduled tasks to avoid resource contention and performance bottlenecks; the optimization and adjustment plan for metadata status may involve checking and maintaining the consistency of data field mapping to ensure the accuracy and completeness of data processing; the optimization and adjustment plan for manual exception handling records may involve providing suggestions for device configuration adjustments, such as expanding the capacity of servers, adding storage devices, etc., or providing optimization suggestions for the overall system configuration to improve the overall performance and stability of the system.

[0121] S205. According to the optimization and adjustment plan, the monitored object is optimized and adjusted, and after the optimization and adjustment of the data acquisition system is completed, the task to be executed is executed to obtain the collected data.

[0122] Among them, performing system optimization before executing the data collection task can reduce the system workload and operation and maintenance pressure. At the same time, it can avoid the failure of data collection task execution due to equipment status, collection and task scheduling configuration, and data conditions, and improve the success rate of data collection.

[0123] However, when the above method involves multiple monitoring objects that need to be adjusted at the same time, the optimization and adjustment schemes of each monitoring object may interfere with each other or be incompatible, resulting in the system being unable to operate normally or reducing efficiency. For example, assuming that the same resource (such as network bandwidth or computing power) needs to be adjusted in the schemes of two monitoring objects, one scheme requires an increase in resource allocation, while the other requires a reduction, which will cause a conflict. In addition, the optimization and adjustment between each monitoring object may be dependent or mutually influential. Adjusting one object may affect the state or performance of another object. For example, adjusting the acquisition frequency of the device may affect the load and speed of data transmission. Therefore, in order to solve these problems, the following describes a method in which the system needs to comprehensively consider the adjustment requirements and mutual relationships of each monitoring object to formulate a final optimization and adjustment scheme to improve the operating efficiency and stability of the system.

[0124] After step S204, the method may further include:

[0125] According to the optimization adjustment plan of each monitored object, determine the final optimization adjustment plan related to the task to be performed in the data acquisition system;

[0126] Accordingly, according to the optimization and adjustment plan, the monitored objects are optimized and adjusted, including:

[0127] According to the final optimization and adjustment plan, each monitored object is optimized and adjusted.

[0128] In this step, you can consider the priority of each task, the resource coordination of the system, the dependencies between the monitored objects and other factors, consider the importance or urgency of the tasks and monitored objects, prioritize the adjustment plans, or reasonably allocate and schedule system resources during the adjustment process to avoid resource competition and conflicts, and analyze the dependencies between the monitored objects to ensure that the adjustment sequence and method will not disrupt the overall function of the system.

[0129] In one example, the optimization adjustment plan for each monitored object is collected and organized; the potential conflicts between the adjustment plans are analyzed, for example, to check whether there are multiple plans to adjust the same resource in different directions, and a detection algorithm or machine learning technology can be selected for automatic identification; the dependencies and mutual influences between the monitored objects are determined, for example, adjusting one object may have an impact on the performance of another object, and an association graph, dependency matrix or machine learning model can be established to perform feature extraction and association learning to determine; priority and weight are assigned to each adjustment plan based on the importance and urgency of the task and the impact on the overall performance of the system; based on the identification of conflicts and associations, the various adjustment plans are integrated to formulate a coordinated final optimization adjustment plan, which may involve modifying or compromising certain plans to ensure the feasibility and effectiveness of the overall plan.

[0130] Among them, in the embodiment of the present application, according to the optimization adjustment scheme of each monitored object, the method for determining the final optimization adjustment scheme related to the task to be performed in the data acquisition system may include:

[0131] Extract features of the optimization adjustment scheme for each monitored object to obtain feature representation of each monitored object;

[0132] Based on the self-attention mechanism and the pre-trained large model, the feature weight corresponding to each monitored object is determined;

[0133] The final optimization adjustment plan is generated based on the feature representation and feature weight of each monitored object.

[0134] In this step, feature extraction may refer to feature extraction of the optimization adjustment plan for each monitored object. Feature extraction may include extracting key parameters, influencing factors, expected effects and other information from the adjustment plan. Since the information corresponding to different monitored objects is different, natural language processing technology or other feature engineering methods can be used to convert this information into a structured feature representation.

[0135] The self-attention mechanism is used to analyze the feature representation of each monitored object. It can help identify and capture the relationship and importance between features. Through the self-attention mechanism, the importance of each feature in the overall scheme is calculated, thereby assigning weights to each feature.

[0136] The pre-trained large model can refer to a multimodal Transformer model that takes the feature representation of all monitored objects as input and is trained with a large amount of historical data. The model can handle different types of data and capture complex patterns and relationships, thereby improving the accuracy of feature weights.

[0137] Therefore, determining the feature weight corresponding to each monitored object based on the output of the self-attention mechanism and the pre-trained large model can more accurately reflect the relative importance of each monitored object in the final optimization adjustment plan.

[0138] The process of comprehensively generating the final optimized adjustment plan based on the feature representation and feature weight of each monitored object can use weighted summation or other aggregation methods to integrate the adjustment plans of various monitored objects into a coordinated overall plan.

[0139] Optionally, the generated final optimization adjustment scheme is verified to ensure its feasibility and effectiveness in practical applications, and the model is iteratively updated using the optimized task execution results and the monitored object status information, so that the generated final optimization adjustment scheme reaches the optimal one.

[0140] By taking advantage of the self-attention mechanism and machine learning models, we can automatically and intelligently formulate optimization and adjustment plans for the data acquisition system to improve the system's operating efficiency and the success rate of task execution.

[0141] Among them, in the embodiment of the present application, according to the final optimization adjustment scheme, the method of optimizing and adjusting each monitoring object may include:

[0142] If the final optimization adjustment plan indicates that manual adjustment is required, the final optimization adjustment plan is sent to the system personnel, and in response to the plan adjustment operation of the system personnel, a manual optimization adjustment plan is obtained, and each monitored object is optimized and adjusted according to the manual optimization adjustment plan;

[0143] If the final optimization adjustment plan indicates that no manual adjustment is required, each monitored object is automatically optimized and adjusted according to the final optimization adjustment plan.

[0144] Among them, it can be determined by predefined rules and conditions whether the final optimization and adjustment scheme characterization needs manual adjustment. These rules and conditions can be set in advance based on factors such as the complexity of the system, the scope of influence of the adjustment, risk assessment, and business needs, and are updated as the system executes tasks. The more exception handling and optimization adjustment schemes there are in the historical information, the fewer schemes that require manual adjustment. For example, when a more complex optimization and adjustment scenario occurs, the system cannot automatically identify it or the historical information shows that no similar problems have been encountered, the scheme is marked as a scheme that requires manual adjustment, and an adjustment prompt is issued to the system personnel, waiting for manual intervention; when the optimization and adjustment scheme is relatively simple or belongs to the common scheme for system exception handling, it is automatically adjusted by the system. For another example, if adjustments involving key business processes or high-risk operations require manual confirmation, then according to business needs and system characteristics, a policy is configured to automatically identify the nature of the adjustment; or risk levels are assigned to different types of adjustments, and high-risk adjustments require manual confirmation; when adjustments involve complex logic or multi-step operations, manual intervention is required to ensure accuracy.

[0145] In one instance, the adjustment of a monitored object involves changing the data collection strategy, which may affect the business process and therefore requires confirmation and adjustment by the user; while adjusting the collection frequency of a device to optimize performance can be done automatically by the system without human intervention.

[0146] Through automated adjustment, the time and energy of human intervention can be reduced, the response speed of the system can be improved, and the system can accurately execute the automatic adjustment plan to reduce the errors that may be caused by human operation. When human judgment is required, system personnel can make adjustments based on actual conditions to ensure the rationality and adaptability of the adjustment plan. Therefore, by combining automation and human intervention, it is ensured that the data acquisition system can be effectively optimized and adjusted under different circumstances.

[0147] Among them, in the embodiment of the present application, the method of executing the task to be executed to obtain the collected data may include:

[0148] Based on the tasks to be executed, data in the source database is collected to obtain initial collected data;

[0149] According to the data conversion rules in the task to be executed, the initial collected data is converted to obtain the collected data.

[0150] Furthermore, based on the tasks to be performed, the system can accurately connect to the source database and use a preset query language, such as SQL (Structured Query Language), to extract the required data, and then extract the relevant tables, fields or records according to the requirements of the tasks to be performed to form the initial collected data; according to the data conversion rules in the tasks to be performed, the initial collected data is processed to obtain the collected data, wherein the data conversion rules may be rules that specify how to convert the initial collected data into a format and structure that meets business requirements and analysis purposes. These rules may include data cleaning, format conversion, aggregate calculations, etc.

[0151] Through a systematic data collection and conversion process, the accuracy, consistency and completeness of the data extracted from the source database are ensured, and the converted data can effectively support business needs and analysis goals.

[0152] In this embodiment of the present application, the method may further include:

[0153] Determine the execution results of the tasks to be executed based on the collected data;

[0154] If the execution result indicates that the task execution failed, the monitoring data information and optimization adjustment plan of the task to be executed will be sent to the system personnel, and the failure analysis results and manual processing plan of the system personnel will be received;

[0155] According to the manual processing plan, re-execute the task to be executed and obtain the collected data after execution;

[0156] The execution process information of the task to be executed is stored, and the execution process information includes monitoring data information, optimization adjustment plan, failure analysis results, and manual processing plan.

[0157] In this step, after collecting data, the system needs to evaluate whether the task is successfully completed, which can be achieved by checking the completeness, accuracy and compliance of the collected data. The execution result can be success or failure, depending on the preset success conditions (such as data volume, data quality, etc.). For example, if the number of collected data items is less than expected, or the data format does not meet the standard, it is judged as execution failure.

[0158] When the execution result shows that the task execution failed, the system will automatically send a notification to the relevant personnel, including the monitoring data information and optimization adjustment plan of the task. The system personnel will analyze the cause of the failure and propose a manual solution to solve the problem. According to the manual solution, the system will re-execute the pending task to obtain new collected data. For example, if the structure of the data source changes, the system personnel may need to update the data conversion rules. The updated data conversion rules are applied and the task is re-executed to obtain the correct data.

[0159] Timely failure handling and re-execution can ensure successful completion of the task. The stored execution process information provides valuable data support for system optimization and troubleshooting. Although manual processing is involved, the complexity and time of manual intervention are reduced through systematic processes and information support.

[0160] The government data collection method provided by the embodiment of the present application significantly improves the efficiency and accuracy of data collection through systematic processes and intelligent technical means. First, by acquiring and classifying tasks to be executed, the system can flexibly handle periodic and one-time tasks; second, the application of monitoring rules ensures real-time monitoring of relevant equipment status, task status and metadata, and generates data analysis results to guide optimization and adjustment through comparison and analysis of regular expressions and parameter thresholds; then, using the self-attention mechanism and pre-trained large models, the system can intelligently integrate the optimization schemes of each monitored object and generate the final optimization adjustment scheme. According to the scheme, the system can automatically or through manual intervention when necessary to ensure the smooth execution of data collection tasks. If the task fails to execute, the system provides detailed monitoring information and adjustment plans, supports manual analysis and processing, and ensures the ultimate success of the task. The information storage of the entire process provides valuable data support for subsequent analysis and optimization, realizes an efficient, reliable and intelligent data collection process, and is suitable for complex government data environments.

[0161] Figure 3 A specific flow chart of a government data collection method provided for this application is as follows: Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, a method for collecting government data is described in detail, and the method includes:

[0162] S01. System users configure data collection tasks, mainly selecting source database tables and target database tables, and configuring corresponding mapping and conversion rules between source and target database tables. They can also configure advanced channel numbers, fault tolerance, and flow, etc.

[0163] S02. After the collection task configuration is completed, if it is a periodic data collection task, the system user configures the execution cycle of data collection according to business needs, such as annual, monthly, daily, hourly, minutely, and secondly execution cycles;

[0164] S03. If it is not a periodic scheduled task, the system user can set a one-time task and trigger a data collection task one by one according to the set one-time scheduling task;

[0165] S04, the system will trigger the set data collection task at the time set in the periodic task according to the periodic scheduled task configured in S02;

[0166] S05. System users set up initial monitoring rule configuration, including monitoring rules for equipment (servers, storage, networks, etc.), collection tasks, scheduling tasks, metadata, manual processing records, etc. The system monitors the equipment status, task status, metadata, and manual processing records according to the monitoring rules, and records the real-time and historical equipment status, task status, metadata, manual processing status, and historical information data. The initial monitoring rules will be automatically optimized and iterated as the system is used. Among them, historical information data can refer to all monitoring data recorded in the system, including current real-time data and historical data;

[0167] S06. System users set initial comparison parameter rules, including devices (servers, storage, networks, etc.), collection tasks, scheduling tasks, metadata, thresholds for manual processing records, and comparison analysis regular expressions. The initial comparison parameter rules will be automatically optimized and iterated as the system is used.

[0168] S07. System users set up optimization and adjustment rules related to initial data collection, including equipment (servers, storage, networks, etc.), collection tasks, scheduling tasks, metadata optimization and adjustment rules, etc. The initial optimization and adjustment rules will be automatically optimized and iterated as the system is used;

[0169] S08. Monitor the device status, task status, metadata status, and manual processing record according to the monitoring rule of S05, and save the monitored device status, task status, metadata status, and manual processing record data into the library table of the monitoring module;

[0170] S09, according to the comparison rules of S06, the storage device status, task status, metadata status, and manual processing record data in the system are compared and analyzed according to the comparison rules, and the comparison and analysis result data is saved in the library table of the analysis module;

[0171] S10, based on the result data analyzed in S09 and the optimization adjustment rules in S07, calculate the equipment optimization adjustment plan, including expanding the server CPU, memory, storage capacity configuration, and expanding the network bandwidth;

[0172] S11. Calculate the task optimization adjustment plan based on the result data analyzed in S09 and the optimization adjustment rules in S07, including the adjustment of the number of concurrent data collection tasks, the staggered scheduling of task times, etc.;

[0173] S12. Calculate the metadata optimization and adjustment plan based on the result data analyzed in S09 and the optimization and adjustment rules in S07, including the impact of metadata on the task, such as checking and maintaining consistency in data field mapping, and affecting execution due to the failure to create a processing table.

[0174] S13, based on the result data analyzed in S09 and the optimization adjustment rules in S07, calculate a manual optimization adjustment plan, including equipment, tasks, metadata and system configuration adjustment plans;

[0175] S14. Based on the single optimization and adjustment plan in S10-S13, use the Transformer architecture of the large model, self-supervised learning, transfer learning and other technologies to comprehensively judge and give the final optimization and adjustment plan;

[0176] S15. If the final optimization and adjustment plan needs to be optimized and adjusted, and manual handling is required, it is transferred to manual optimization and adjustment. The system user intervenes to analyze various optimization and adjustment plans and decide which optimization and adjustment plan to adopt in the end. This scenario is suitable for more complex optimization and adjustment scenarios. When the system cannot automatically determine, manual intervention is required;

[0177] S16. If the final optimization and adjustment plan needs to be optimized and adjusted, and no manual processing is required, the final optimization and adjustment plan obtained after the comprehensive analysis of S14 is automatically executed;

[0178] S17. If no optimization adjustment is required or S15 and S16 are completed, the work of reading data from the source database is started, and the data in the source database is read into the data executor;

[0179] S18, according to the configured data acquisition data conversion rules, the source data read in S17 is converted to form converted result data;

[0180] The result data converted in S19 and S18 is written into the target database;

[0181] S20, monitor the data collection tasks and scheduling tasks, monitor the execution details of the tasks, record the detailed logs of the tasks in the library table, record the abnormal analysis results in the library table, and present the analysis results to the system;

[0182] S21. If the task is completed normally, the system user analyzes the cause of task failure based on the task monitoring records and formulates a manual processing plan for task exception handling;

[0183] S22. Execute the manual processing solution for the task exception handling in S21. The manual processing solution provides an experience record of automatic optimization for subsequent iterative optimization, provides training data to subsequent monitoring, analysis, and optimization modules as training data, and optimizes and improves subsequent data collection and prediction optimization devices.

[0184] The government data collection method provided in the embodiment of the present application introduces a data collection pre-judgment optimization device before the data collection scheduling task is triggered and the data collection task is executed, and pre-judgment analysis and processing of various situations that may cause data collection failure are carried out in advance, thereby reducing the probability of data collection failure and improving the success rate of data collection. It has a positive impact on the business processes that rely on these data for processing analysis, in-depth mining and other decision-making, and improves the operating efficiency of government data. The introduced data collection pre-judgment optimization device pre-places data collection exception processing, that is, when potential problems occur in the data collection task, the system can automatically warn and analyze and take pre-optimization disposal measures. This proactive processing method is not only efficient, but also can repair problems in advance, avoiding the impact of task failure on the overall collection efficiency. At the same time, data collection exception processing has been advanced to before the data collection task is executed, and the system will issue an advance pre-judgment optimization method, which can reduce workload and operation and maintenance pressure, and reduce the technical requirements for operation and maintenance personnel. Moreover, since the data collection task exception handling is highly automated, the probability of manual analysis, diagnosis and repair by manpower is greatly reduced, thereby reducing operating costs. The entire process of each data collection exception handling will be recorded by the monitoring module, and the relevant records will be automatically applied to the subsequent data collection prediction and optimization device. These exception handling experiences will be referenced and reused in the subsequent comprehensive prediction module.

[0185] Figure 4 A schematic diagram of the structure of a government data collection device provided for this application, such as Figure 4 As shown, the government data collection device 40 provided in this embodiment includes:

[0186] The acquisition module 401 is used to acquire tasks to be executed;

[0187] Monitoring module 402, used to monitor the monitoring objects that affect the execution of the task to be executed, and generate monitoring data information of each monitoring object. The monitoring objects include the equipment status, task status, metadata and manual exception processing records related to the task to be executed in the data acquisition system. The monitoring rules include the specific type and monitoring parameters of the monitoring object;

[0188] The analysis module 403 is used to analyze the monitoring data information of each monitored object according to the comparison parameter rule to obtain a data analysis result, where the comparison parameter rule is a regular expression corresponding to the monitored object and a monitoring parameter threshold;

[0189] A determination module 404 is used to determine an optimization adjustment plan for the monitored object according to the data analysis results;

[0190] The processing module 405 is used to optimize and adjust the monitored object according to the optimization and adjustment plan, and after the optimization and adjustment of the data acquisition system is completed, execute the task to be executed to obtain the collected data.

[0191] In a possible implementation, the acquisition module 401 may also be used to:

[0192] Responding to the configuration operation of the system personnel, obtaining the data collection task;

[0193] Determine the scheduling task type in the data collection task;

[0194] If the scheduling task type is a periodic task, determine the execution cycle of the data collection task and the tasks to be executed corresponding to the execution cycle;

[0195] If the scheduling task type is a one-time task, the data collection task is determined as a task to be executed.

[0196] In a possible implementation, the analysis module 403 may also be used to:

[0197] According to the comparison parameter rules, determine the regular expression and monitoring parameter threshold corresponding to the monitoring object;

[0198] Based on the regular expression, a matching analysis is performed on the monitoring data information of the monitoring object to determine a matching analysis result, where the matching analysis result is used to indicate whether the monitoring data information includes a preset monitoring parameter;

[0199] If the matching analysis result indicates that the monitoring information includes preset monitoring parameters, the preset monitoring parameters are compared and analyzed with the parameter thresholds respectively corresponding to the monitoring parameter thresholds to determine the comparison analysis result, which indicates whether the monitoring parameters of the monitored object exceed the monitoring parameter thresholds;

[0200] The data analysis results are generated according to the matching analysis results and the comparison analysis results.

[0201] In a possible implementation, the determination module 404 may also be used to:

[0202] According to the optimization adjustment plan of each monitored object, determine the final optimization adjustment plan related to the task to be performed in the data acquisition system;

[0203] Accordingly, according to the optimization and adjustment plan, the monitored objects are optimized and adjusted, including:

[0204] According to the final optimization and adjustment plan, each monitored object is optimized and adjusted.

[0205] In a possible implementation, the determination module 404 may also be used to:

[0206] Extract features of the optimization adjustment scheme for each monitored object to obtain feature representation of each monitored object;

[0207] Based on the self-attention mechanism and the pre-trained large model, the feature weight corresponding to each monitored object is determined;

[0208] The final optimization adjustment plan is generated based on the feature representation and feature weight of each monitored object.

[0209] In a possible implementation, the determination module 404 may also be used to:

[0210] If the final optimization adjustment plan indicates that manual adjustment is required, the final optimization adjustment plan is sent to the system personnel, and in response to the plan adjustment operation of the system personnel, a manual optimization adjustment plan is obtained, and each monitored object is optimized and adjusted according to the manual optimization adjustment plan;

[0211] If the final optimization adjustment plan indicates that no manual adjustment is required, each monitored object is automatically optimized and adjusted according to the final optimization adjustment plan.

[0212] In a possible implementation, the determination module 404 may also be used to:

[0213] Based on the tasks to be executed, data in the source database is collected to obtain initial collected data;

[0214] According to the data conversion rules in the task to be executed, the initial collected data is converted to obtain the collected data.

[0215] In a possible implementation, the processing module 405 may also be used to:

[0216] Determine the execution results of the tasks to be executed based on the collected data;

[0217] If the execution result indicates that the task execution failed, the monitoring data information and optimization adjustment plan of the task to be executed will be sent to the system personnel, and the failure analysis results and manual processing plan of the system personnel will be received;

[0218] According to the manual processing plan, re-execute the task to be executed and obtain the collected data after execution;

[0219] The execution process information of the task to be executed is stored, and the execution process information includes monitoring data information, optimization adjustment plan, failure analysis results, and manual processing plan.

[0220] The government data collection device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, so this embodiment will not be described in detail here.

[0221] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 5As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0222] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0223] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0224] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0225] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0226] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0227] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0228] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0229] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0230] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0231] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0232] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0234] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0235] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0236] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for collecting government data, characterized in that: Applied to a data acquisition system, the method comprises: Get the tasks to be executed; According to the monitoring rules, the monitoring objects that affect the execution of the task to be executed are monitored to generate monitoring data information of each monitoring object, wherein the monitoring objects include the equipment status, task status, metadata and manual exception handling records related to the task to be executed in the data acquisition system, and the monitoring rules include the specific type and monitoring parameters of the monitoring objects; For each monitored object, the monitored data information is analyzed according to a comparison parameter rule to obtain a data analysis result, wherein the comparison parameter rule is a regular expression corresponding to the monitored object and a monitoring parameter threshold; Determine an optimization adjustment plan for the monitored object according to the data analysis results; According to the optimization and adjustment scheme, the monitored object is optimized and adjusted, and after the optimization and adjustment of the data acquisition system is completed, the task to be executed is executed to obtain the collected data.

2. The method according to claim 1, characterized in that The obtaining of tasks to be executed includes: Responding to the configuration operation of the system personnel, obtaining the data collection task; Determining the scheduling task type in the data collection task; If the scheduling task type is a periodic task, determining the execution cycle of the data collection task and the tasks to be executed corresponding to the execution cycle; If the scheduling task type is a one-time task, the data collection task is determined as a task to be executed.

3. The method according to claim 1, characterized in that The monitoring data information is analyzed according to the comparison parameter rules to obtain data analysis results, including: Determine, according to the comparison parameter rule, a regular expression and a monitoring parameter threshold value corresponding to the monitoring object; Based on the regular expression, performing a matching analysis on the monitoring data information of the monitoring object to determine a matching analysis result, wherein the matching analysis result is used to indicate whether the monitoring data information includes a preset monitoring parameter; If the matching analysis result indicates that the monitoring information includes preset monitoring parameters, the preset monitoring parameters are compared and analyzed with the parameter thresholds respectively corresponding to the monitoring parameter thresholds to determine a comparison analysis result, wherein the comparison analysis result indicates whether the monitoring parameters of the monitored object exceed the monitoring parameter thresholds; A data analysis result is generated according to the matching analysis result and the comparison analysis result.

4. The method according to claim 1, characterized in that After determining the optimization adjustment plan of the monitored object according to the data analysis result, the method further includes: Determine, according to the optimization adjustment plan of each monitored object, a final optimization adjustment plan related to the task to be performed in the data acquisition system; Accordingly, the optimizing and adjusting the monitored object according to the optimizing and adjusting scheme includes: According to the final optimization and adjustment plan, each monitoring object is optimized and adjusted.

5. The method according to claim 4, characterized in that Determining the final optimization adjustment scheme related to the task to be performed in the data acquisition system according to the optimization adjustment scheme of each monitored object includes: Extract features of the optimization adjustment scheme for each monitored object to obtain feature representation of each monitored object; Based on the self-attention mechanism and the pre-trained large model, the feature weight corresponding to each monitored object is determined; The final optimization adjustment plan is generated based on the feature representation and feature weight of each monitored object.

6. The method according to claim 4, characterized in that The optimizing and adjusting each monitoring object according to the final optimization and adjustment scheme includes: If the final optimization adjustment scheme indicates that manual adjustment is required, the final optimization adjustment scheme is sent to the system personnel, and in response to the scheme adjustment operation of the system personnel, a manual optimization adjustment scheme is obtained, and each monitored object is optimized and adjusted according to the manual optimization adjustment scheme; If the final optimization adjustment scheme indicates that manual adjustment is not required, each monitored object is automatically optimized and adjusted according to the final optimization adjustment scheme.

7. The method according to claim 6, characterized in that The executing the task to be executed to obtain the collected data includes: Based on the task to be executed, data in the source database is collected to obtain initial collected data; According to the data conversion rule in the task to be executed, the initial collected data is converted to obtain the collected data.

8. The method according to claim 1, characterized in that The method further comprises: Determining the execution result of the task to be executed according to the collected data; If the execution result indicates that the task execution has failed, the monitoring data information and optimization adjustment plan of the task to be executed are sent to the system personnel, and the failure analysis result and manual processing plan of the system personnel are received; According to the manual processing scheme, re-execute the task to be executed to obtain the collected data after execution; The execution process information of the task to be executed is stored, wherein the execution process information includes monitoring data information, optimization adjustment plan, failure analysis result, and manual processing plan.

9. A government data collection device, characterized in that: include: The acquisition module is used to obtain tasks to be executed; A monitoring module, used to monitor the monitoring objects that affect the execution of the tasks to be executed, and generate monitoring data information of each monitoring object, wherein the monitoring objects include the equipment status, task status, metadata and manual exception handling records related to the tasks to be executed in the data acquisition system, and the monitoring rules include the specific types and monitoring parameters of the monitoring objects; An analysis module is used to analyze the monitoring data information of each monitored object according to a comparison parameter rule to obtain a data analysis result, wherein the comparison parameter rule is a regular expression corresponding to the monitored object and a monitoring parameter threshold; A determination module, used to determine an optimization adjustment plan for the monitored object according to the data analysis result; The processing module is used to optimize and adjust the monitored object according to the optimization and adjustment plan, and after the optimization and adjustment of the data acquisition system is completed, execute the task to be executed to obtain the collected data.

10. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 8.

11. A data acquisition system, characterized in that: It includes a data acquisition prediction optimization device and a data acquisition device; The data acquisition prediction and optimization device is used to monitor and optimize the monitored object; The data acquisition device is used to execute the task to be executed to obtain the collected data.

12. The system according to claim 11, characterized in that The data acquisition prediction and optimization device includes a monitoring module, an analysis module and an optimization module; The monitoring module is used to obtain the tasks to be executed, and according to the monitoring rules, monitor the monitoring objects that affect the execution of the tasks to be executed, and generate monitoring data information of each monitoring object. The monitoring objects include the equipment status, task status, metadata and manual exception handling records related to the tasks to be executed, and the monitoring rules include the specific types and monitoring parameters of the monitoring objects; The analysis module is used to analyze the monitoring data information of each monitored object according to the comparison parameter rule to obtain a data analysis result, wherein the comparison parameter rule is a regular expression and a monitoring parameter threshold corresponding to the monitored object, and determine an optimization adjustment plan for the monitored object according to the data analysis result; The optimization module is used to optimize and adjust the monitored object according to the optimization and adjustment plan.

Images