Water conservancy monitoring data multi-index and multi-dimension statistical method based on spring batch

Abstract

The invention relates to the technical field of data processing, and discloses a spring batch-based water conservancy monitoring data multi-index and multi-dimensional statistical method, computer equipment, a computer readable storage medium and a computer program product in order to solve the problem of low efficiency of an existing water conservancy monitoring data statistical method. The method comprises the following steps: acquiring configuration information of at least one water conservancy monitoring data index to be counted, wherein the configuration information comprises configuration parameters and a statistical time range; querying a water conservancy monitoring database according to the configuration information to obtain corresponding result data; and processing the result data through the batch processing framework, and storing the processed data as a general object into a corresponding statistical target table. By adopting the method, multi-index, multi-dimension and multi-type statistics of the water conservancy monitoring data can be realized, the statistical efficiency of the water conservancy monitoring data is improved, and the development workload is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of data processing, and more specifically, to a statistical method, a computer device, a computer-readable storage medium, and a computer program product for multi-index and multi-dimension of water conservancy monitoring data based on spring batch. Background Art

[0002] In the water conservancy industry, there are various types of real-time water conservancy monitoring data, such as rainfall monitoring data, river channel water level and flow monitoring data, water quality index monitoring data, pre-gate water level, post-gate water level, opening degree of the gate, etc. These water conservancy monitoring data need to be statistically calculated for average value, maximum value, cumulative value, etc. according to different time dimensions to provide intuitive statistical data for users to help them make judgments and decisions.

[0003] Currently, the water conservancy monitoring data is usually statistically calculated in a hard-coded manner, that is, combined with the business requirements for the statistical calculation of water conservancy monitoring data, the data within the corresponding index time range of the water conservancy monitoring data is obtained from the corresponding data source (usually a table in a relational database), and then hard-coded according to the corresponding statistical dimension and statistical type. However, the data sources of different types of water conservancy monitoring data are different, and certain adaptations are required for different projects and different indicators. If ten indicators are to be statistically calculated, ten corresponding hard-coded writing logics need to be implemented. Moreover, a large number of database reads and writes will also cause heavy database load and poor performance. In addition, if the requirements for the statistical dimensions and indicators of water conservancy monitoring data change or the data volume increases, the code will change and it will be unable to quickly and timely respond to user requirements. Therefore, implementing the statistical calculation of water conservancy monitoring data through hard coding not only has a huge workload and poor efficiency, but also has the defects of extremely poor scalability and difficult maintenance.

[0004] Another statistical method is to use the frameworks in the big data field to implement the statistical calculation of water conservancy monitoring data. These frameworks have different advantages and applicable scenarios when processing large-scale data sets. The above big data frameworks are all distributed or stream processing frameworks, suitable for basic statistical analysis in batch processing mode, such as calculating the maximum value, minimum value, average value, etc. However, there are still the following defects: (1) Introducing the relevant frameworks in the big data field in the project requires introducing more service deployments and operations and maintenance, making the maintenance and problem troubleshooting of the project difficult and increasing the project development and operation and maintenance costs; (2) Using the relevant frameworks in the big data field requires professional personnel to develop, or the current project personnel to develop after learning, which will lengthen the development cycle related to the statistical calculation of water conservancy monitoring data and increase the learning and maintenance costs; (3) After using the frameworks in the big data field for development, it is also necessary to interface with the project itself, making the implementation process chain of the statistics cumbersome and introducing complexity to subsequent use. Summary of the Invention

[0005] To solve the problem of low efficiency in the existing statistical methods for water conservancy monitoring data, the present invention provides a statistical method, a computer device, a computer-readable storage medium, and a computer program product for multi-index and multi-dimension of water conservancy monitoring data based on spring batch, which can perform statistics on multi-index, multi-dimension, and multi-type of water conservancy monitoring data, reducing the development workload and improving the statistical efficiency of water conservancy monitoring data.

[0006] By adopting this method, the following can be achieved. To achieve the above object, according to the first aspect of the present invention, a statistical method for multi-index and multi-dimension of water conservancy monitoring data based on spring batch is provided, and the method includes:

[0007] Obtain the configuration information of at least one water conservancy monitoring data index to be statistically analyzed, where the configuration information includes configuration parameters and the statistical time range;

[0008] Query the water conservancy monitoring database according to the configuration information to obtain the corresponding result data;

[0009] Process the result data through a batch processing framework and store the processed data as a general object into the corresponding statistical target table.

[0010] Further, the above statistical method for multi-index and multi-dimension of water conservancy monitoring data based on spring batch further includes setting multiple configuration items according to the water conservancy monitoring data index to be statistically analyzed. The configuration items include the source table corresponding to the water conservancy monitoring data index, the station code field, the corresponding code and name of the index, the statistical dimension, the statistical type, the last statistical time, and the statistical target table; forming a list of water conservancy monitoring data configuration items with multiple configuration items and storing it in the water conservancy monitoring database.

[0011] Further, obtaining the configuration information of at least one water conservancy monitoring data index to be statistically analyzed includes starting a custom timed task; obtaining the configuration information of at least one water conservancy monitoring data index to be statistically analyzed from the list of water conservancy monitoring data configuration items through the timed task.

[0012] Further, querying the water conservancy monitoring database according to the configuration information to obtain the corresponding result data includes generating a structured query language statement according to the configuration information; querying the water conservancy monitoring database according to the structured query language statement to obtain the corresponding result data in the source table.

[0013] Further, the result data is processed through a batch processing framework, and the processed data is stored in the corresponding statistical target table as a general object, including initiating a task through a task launcher in the batch processing framework, passing the configuration parameters and the statistical time range included in the result data to the task object through task parameters, and having the task steps perform statistics on the configuration information of each water conservancy monitoring data indicator, and storing the statistically processed data in the corresponding statistical target table as a general object.

[0014] Further, the task steps perform statistics on the configuration information of each water conservancy monitoring data indicator, and store the statistically processed data in the corresponding statistical target table as a general object, including the task steps entering a partitioner, where the partitioner partitions according to the statistical time range and statistical dimension obtained by the task steps, obtains at least one time range parameter under the same dimension, and passes the at least one time range parameter to a data reader; the data reader dynamically assembles a structured query language statement to query the original data from the source table according to the configuration items corresponding to the at least one time range parameter, and passes the queried original data to a data processor; the data processor processes the queried original data according to the statistical type, and passes the processed data to a data writer; the data writer periodically submits the processed data to the corresponding statistical target table in the water conservancy monitoring database according to the batch processing size set by the task steps.

[0015] Further, the data writer periodically submits the processed data to the corresponding statistical target table in the water conservancy monitoring database according to the batch processing size set by the task steps, including the data writer receiving the data processed by the data processor; determining the set data volume size according to the batch processing size set by the task steps; and whenever the processed data accumulates to the set data volume size, submitting the processed data as a general object to the specified statistical target table in the water conservancy monitoring database.

[0016] According to the second aspect of the present invention, there is also provided a computer device, which includes a memory, a processor, and a computer program stored on the memory, and the processor executes the computer program to implement the steps of any one of the above methods.

[0017] According to the third aspect of the present invention, there is also provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of any one of the above methods are implemented.

[0018] According to the fourth aspect of the present invention, there is also provided a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any one of the above methods are implemented.

[0019] Generally speaking, compared with the prior art, the above technical solution conceived by the present invention can achieve the following beneficial effects:

[0020] A statistical method for multi-index and multi-dimension of water conservancy monitoring data based on spring batch provided by the present invention can obtain data from the source table according to the configuration information by acquiring the configuration information of the water conservancy monitoring data indicators, and then process the data through the spring batch batch processing framework to obtain data that can be used as a general object, and store it in the corresponding statistical target table, that is, it can realize storing the data of different source tables into the statistical target table with the same table structure as the general object, so as to realize one-time development compatible with all indicators, all dimensions and all types of statistics, realize the statistics of multi-index, multi-dimension and multi-type of water conservancy monitoring data, and improve the statistical efficiency of water conservancy monitoring data. Moreover, the configuration items of different indicators can be added, modified or deleted according to the actual situation without re-coding, so the development workload is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0022] Figure 1 It is a schematic diagram of the basic working principle of the spring batch framework;

[0023] Figure 2 It is a schematic flow diagram of a statistical method for multi-index and multi-dimension of water conservancy monitoring data based on spring batch provided by an embodiment of the present application;

[0024] Figure 3 It is a schematic flow diagram of a statistical method for multi-index and multi-dimension of water conservancy monitoring data based on spring batch provided by another embodiment of the present application;

[0025] Figure 4 It is a schematic internal structure diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following further describes the present invention in detail with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0027] The terms "first", "second", "third", etc. in the description, claims and the above-mentioned drawings of this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0028] This embodiment provides a statistical method for multi-index and multi-dimension of water conservancy monitoring data based on spring batch. Spring batch is a framework for batch processing, used for batch processing tasks of a large amount of data, such as data import, data export, report generation, etc.

[0029] The basic components and basic working principle of the spring batch framework are as Figure 1 shown. The spring batch framework includes components such as JobRepository (task manager), JobLauncher (task launcher), Job (task object configured based on the framework), Step (step object in the task, abbreviated as task step), ItemReader (data reader), ItemProcessor (data processor), ItemWriter (data writer), etc. In addition, it also includes JobParameters (task parameters), which are passed when the task is started; and Partitioner, which can be partitioned in the Step and executed in parallel.

[0030] Table 1 is a list of water conservancy monitoring data configuration items. Referring to Table 1, the water conservancy monitoring data to be statistically analyzed comes from different data monitoring tables (also called source tables), and each table has one or more indicators to be statistically analyzed. The statistical dimensions are respectively hour, day, month, etc., and the statistical types are respectively sum, average value or maximum value, etc.

[0031] Table 1 List of Water Conservancy Monitoring Data Configuration Items

[0032]

[0033] Therefore, in the configuration method, relevant configurations are first made for the water conservancy monitoring data to be counted. The configuration items include the source table corresponding to the water conservancy monitoring data index, the station code field, the corresponding code and name of the index, the statistical dimension, the statistical type, the last statistical time, the statistical target table, etc.; the configuration items are stored in the water conservancy monitoring database to obtain a list of water conservancy monitoring data configuration items, which is convenient to query the configuration items from the list of water conservancy monitoring data configuration items each time statistics are performed. The configuration items are editable, including modification, addition or deletion. In the case of modification, addition or deletion of the configuration of the water conservancy monitoring data to be counted, the latest configuration can be obtained.

[0034] Then, SQL (Structured Query Language) statements can be automatically generated according to the configuration. Data is obtained from the source table through the SQL statements. For example, a virtual SQL statement is assembled according to the configuration of the first row of data in Table 1 as follows: SELECT drp FROM st_pptn_r WHERE stcd=?AND tm BETWEEN 2024 / 12 / 19 10:30 AND 2024 / 12 / 20 10:30. Then, after processing the queried data through spring batch, the result data is stored in the specified statistical target table using a general object (the table structure data of the general object is shown in Table 2), that is, the table where the statistical data for different data source tables is finally stored, and the table structures of all statistical target tables and general objects are the same. Therefore, this embodiment can develop once to be compatible with all indexes, all dimensions and all types of statistics, and only needs to add, modify or delete the configuration to meet the requirement changes, greatly reducing the development workload.

[0035] Table 2 Structure data table of general object

[0036] Table Field Field Type Field Description id bigint Primary Key id stcd varchar Station Code tm datetime Time index_column varchar Index Parameter index_name varchar Index Name value decimal Statistical Value type varchar Statistical Type year varchar Year month varchar Month day varchar Day hour varchar Hour dimension varchar Dimension

[0037] As Figure 2 shown, a statistical method for multi-index and multi-dimension of water conservancy monitoring data based on spring batch is provided. This method can be executed by a terminal or by a server that communicates with the terminal through a network. Among them, the terminal can be but is not limited to various personal computers, laptops, smartphones, tablets, Internet of Things devices, etc. The server can be an independent server or implemented by a server cluster composed of multiple servers. Taking the application of this method to the terminal as an example, the following steps are included:

[0038] Step 201, obtain the configuration information of at least one water conservancy monitoring data index to be counted, and the configuration information includes configuration parameters and the statistical time range.

[0039] Among them, the configuration parameters include the index name, the corresponding source table name, etc.

[0040] Step 202: Query the water conservancy monitoring database according to the configuration information to obtain the corresponding result data.

[0041] Step 203: Process the result data through a batch processing framework, and store the processed data as a general object into the corresponding statistical target table.

[0042] Among them, the batch processing framework can adopt spring batch.

[0043] Before step 201, the above method further includes setting multiple configuration items according to the water conservancy monitoring data indicators to be statistically analyzed. The configuration items include the source table corresponding to the water conservancy monitoring data indicators, the station code field, the corresponding code and name of the indicators, the statistical dimension, the statistical type, the last statistical time, and the statistical target table; forming a list of water conservancy monitoring data configuration items with multiple configuration items, and storing them in the water conservancy monitoring database.

[0044] Through the configuration method of this embodiment, the purpose of multi-index, multi-dimension, and multi-type statistical analysis of water conservancy monitoring data can be achieved, the statistical efficiency of water conservancy monitoring data is improved, and the development workload is reduced.

[0045] Such as Figure 3 shown, in one embodiment, a multi-index and multi-dimension statistical method for water conservancy monitoring data based on spring batch is further provided, which specifically includes the following steps:

[0046] Step 1: Perform relevant configuration on the water conservancy monitoring data indicators to be statistically analyzed. The configuration items include the source table corresponding to the water conservancy monitoring data indicators, the station code field corresponding to the indicators, the time field corresponding to the indicators, the corresponding code and name of the indicators, the statistical dimension of the indicators, the statistical type of the indicators, the last statistical time of the indicators, the statistical target table of the indicators, etc. If not all of Table 1 is included, store the configured statistical configuration of the water conservancy monitoring data indicators in the database to facilitate querying the configuration from the table every time statistics are performed. If there are modifications, additions, deletions, etc. to the configuration, it can ensure that the latest configuration is obtained.

[0047] Step 2: After the statistical configuration of the water conservancy monitoring data is completed, start a custom scheduled task. First, the scheduled task will obtain the configuration data from the configuration table. After obtaining the configuration corresponding to the indicators, determine the statistical time range according to the last statistical time of the indicators. If statistics have never been performed, obtain the earliest time of the data in the source table according to the source table, and form the statistical time range according to the current time. Then execute the subsequent steps to calculate and process the statistical indicators of the original data.

[0048] Step 3: Initiate a task in Spring Batch through a JobLauncher, and pass the configuration parameters and time range in the obtained data statistics configuration to the Job object through JobParameters, so that the corresponding parameters can be obtained and used when the Job is executed. After the task starts execution, it first enters a task step (Step). At this time, it is assumed that a configuration item has been passed into the Step. All the following operations use the data statistics operation of a single configuration item as an example.

[0049] Step 4: The Step first enters a partitioner configured according to the characteristics of Spring Batch. The partitioner will partition by month, day, hour, etc. according to the time range obtained by the Step and the statistical dimension, that is, pass the time range parameters under the same dimension into different partitions. After partitioning, the parameters of each partition will be placed in the context of the Spring Batch Step and passed to the next step of the Step. At this time, to improve the statistical efficiency of different partitions, parallel execution and multi-threaded execution of partitions can be set.

[0050] Step 5: After the partitioned data reader (ItemReader) obtains the time range configuration according to the context information, as well as fields such as the source table, station code field, time field, index field, and statistical type in the configuration, it dynamically assembles an SQL statement to query the original data from the source table, as described above. At this time, the data obtained according to the time range parameters in the above partition is all the data in the required statistical dimension; at this time, functions such as allowing the ItemReader to skip some records, retry on failure, limit the number of failure attempts, and abnormal exit can be set. Using the ItemReader to query the original data is to narrow the time range to avoid performance bottlenecks caused by querying a large amount of data and the memory pressure caused by passing a large amount of data in the task.

[0051] Step 6: After the above ItemReader reads the data, it passes it to the data processor (ItemProcessor). In the ItemProcessor, according to its own data processing logic, in the statistics of water conservancy monitoring data, this step is to perform calculations according to the statistical type, and then process or transform the data. Then the ItemProcessor will pass the processed and assembled data to the data writer (ItemWriter).

[0052] Step 7: The ItemWriter periodically submits the data passed by the ItemProcessor to the database in one go when the accumulated data reaches the batch size set in the Step, reducing the pressure on the database caused by multiple connections and writes. The ItemWriter can also be set to skip some records, retry failures, limit the number of failure retries, and be compatible with abnormal exits.

[0053] In this embodiment, the provided statistical method for multi-index and multi-dimension of water conservancy monitoring data based on spring batch has the following beneficial effects:

[0054] (1) The statistical method of water conservancy monitoring data can be configured through indicators. By directly obtaining the indicator configuration, the water conservancy monitoring data of different indicators can be configured and modified according to the actual situation. Adding or modifying statistical indicators does not require recoding, but only requires modifying the configuration.

[0055] (2) The implementation logic of water conservancy monitoring data statistics is to directly query by dynamically concatenating SQL parameters to form an SQL statement (as shown in the above example: SELECT drp FROM st_pptn_r WHERE stcd =? AND tm BETWEEN 2024 / 12 / 19 10:30 AND 2024 / 12 / 20 10:30), and directly return the source table data under the corresponding dimension, effectively reducing the pressure on the database caused by a large amount of data.

[0056] (3) The statistical time range is automatically determined and obtained by the system. Whether it is full data, that is, the indicator data that has not been statistically processed, or incremental data, that is, the indicator data that has been statistically processed, can be recognized during statistics and queried within the specified time range, avoiding repeated statistical calculations for the already statistically processed data and consuming a large amount of resources.

[0057] (4) The general statistical solution implemented based on the framework can read in parallel and write in batches, and also includes operations such as logging, tracing, transactions, task restart, skipping, duplication, and resource management, which can avoid performance defects while ensuring efficiency.

[0058] (5) The general statistical solution implemented based on the framework only needs to be implemented once and can be used for all indicators, all dimensions, and all types of statistics. Based on configuration, changes in requirements can be realized conveniently and quickly.

[0059] This application also provides a computer device, and its internal structure diagram can be as Figure 4As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a statistical method for multi-index and multi-dimension of water conservancy monitoring data based on spring batch.

[0060] Those skilled in the art can understand that Figure 4 the structure shown in the figure is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0061] As Figure 4 shown, the present application also provides a computer device, which includes a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps in the above method embodiments.

[0062] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, it implements the steps in the above method embodiments. Among them, the computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, micro drives, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nano-systems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0063] The present application also provides a computer program product, including a computer program. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0064] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, some steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0065] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0066] As described above, the above are only exemplary embodiments of the present disclosure, and the scope of the present disclosure cannot be limited thereby. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure still fall within the scope covered by the present disclosure. Those skilled in the art will readily think of the implementation schemes of the present disclosure after considering the specification and practicing the present disclosure here. This application aims to cover any variations, uses or adaptive changes of the present disclosure, and these variations, uses or adaptive changes follow the general principles of the present disclosure and include the common general knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

[0067] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered that the scope described in this specification is covered.

[0068] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A statistical method for multi-index and multi-dimension of water conservancy monitoring data based on spring batch, characterized in that, Including: Obtain the configuration information of at least one water conservancy monitoring data indicator to be counted, where the configuration information includes configuration parameters and the statistical time range; Query the water conservancy monitoring database according to the configuration information to obtain the corresponding result data; Process the result data through a batch processing framework and store the processed data as a general object into the corresponding statistical target table.

2. The method according to claim 1, characterized in that, The method further includes: Set multiple configuration items according to the water conservancy monitoring data indicators to be counted. The configuration items include the source table corresponding to the water conservancy monitoring data indicator, the station code field, the corresponding code and name of the indicator, the statistical dimension, the statistical type, the last statistical time, and the statistical target table; Form the multiple configuration items into a water conservancy monitoring data configuration item list and store it in the water conservancy monitoring database.

3. The method according to claim 2, wherein The obtaining of the configuration information of at least one water conservancy monitoring data indicator to be counted includes: Start a custom timed task; Obtain the configuration information of at least one water conservancy monitoring data indicator to be counted from the water conservancy monitoring data configuration item list through the timed task.

4. The method according to claim 2, wherein The querying of the water conservancy monitoring database according to the configuration information to obtain the corresponding result data includes: Generate a structured query language statement according to the configuration information; Query the water conservancy monitoring database according to the structured query language statement and obtain the corresponding result data in the source table.

5. The method according to claim 2, characterized in that The processing of the result data through a batch processing framework and storing the processed data as a general object into the corresponding statistical target table includes: In the batch processing framework, initiate a task through a task launcher, pass the configuration parameters and the statistical time range included in the result data to the task object through task parameters, and let the task steps count the configuration information of each water conservancy monitoring data indicator, and store the counted data as a general object into the corresponding statistical target table.

6. The method according to claim 5, characterized in that, The task steps count the configuration information of each water conservancy monitoring data indicator and store the counted data as a general object into the corresponding statistical target table, including: The task steps enter the partitioner, and the partitioner partitions according to the statistical time range and statistical dimension obtained by the task steps to obtain at least one time range parameter under the same dimension, and passes the at least one time range parameter to the data reader; The data reader dynamically assembles a structured query language statement to query the original data from the source table according to the configuration item corresponding to the at least one time range parameter, and passes the queried original data to the data processor; The data processor processes the queried original data according to the statistical type and passes the processed data to the data writer; The data writer periodically submits the processed data to the corresponding statistical target table in the water conservancy monitoring database according to the batch processing size set by the task steps.

7. The method according to claim 6, wherein The data writer periodically submits the processed data to the corresponding statistical target table in the water conservancy monitoring database according to the batch processing size set by the task steps, including: The data writer receives the data processed by the data processor; Determine the set data volume size according to the batch processing size set by the task steps; Whenever the accumulated processed data reaches the set data volume, the processed data is used as a general object and submitted to the specified statistical target table in the water conservancy monitoring database.

8. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, A computer program is stored thereon. When the computer program is executed by a processor, the steps of the method according to any one of claims 1-7 are implemented.

10. A computer program product, characterized in that, It includes a computer program. When the computer program is executed by a processor, the steps of the method according to any one of claims 1-7 are implemented.

Images