Data processing method and system of water affair management system

Abstract

The invention belongs to the technical field of water affair data management. The invention provides a data processing method and system of a water affair management system. The method comprises the following steps: S1, optimizing a data processing flow: introducing a parallel processing mechanism, splitting a water affair data processing task into a plurality of subtasks which can be executed in parallel, and dynamically adjusting task allocation according to a real-time state of system resources by utilizing a dynamic scheduling algorithm; s2, strengthening data cleaning and conversion capability: adopting a data cleaning tool based on an intelligent data cleaning algorithm to automatically identify and process various types of water affair data errors; optimizing the water affair data conversion and integration process by using a machine learning model conversion engine; and S3, data integration: constructing a unified data integration platform, wherein the data integration platform can automatically identify and analyze data formats and structures from different data sources. According to the invention, comprehensive optimization and performance improvement of the water affair data processing flow are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of water affairs data management, and in particular to a data processing method and system for a water affairs management system. Background Art

[0002] With the rapid development of information technology, the importance of data center as the core hub of enterprise data management has become increasingly prominent. By integrating data resources from various business systems, data center provides enterprises with unified data services to support data analysis, business decision-making, product innovation and other aspects. However, in the face of massive, heterogeneous and complex data environments, existing data processing methods face many challenges, such as low processing efficiency, difficulty in ensuring data accuracy, and excessive resource consumption.

[0003] It is mainly reflected in the following aspects:

[0004] 1. Data processing flow

[0005] The existing data processing process of the data center usually includes data collection, storage, processing, analysis and application. Among these links, data processing is the most core and challenging part. Traditional data processing methods often use a sequential execution method to gradually clean, convert and integrate data. This method is particularly inefficient when processing large-scale data sets. In addition, each link in the data processing process often lacks an effective coordination mechanism, which makes it difficult to further improve the efficiency of data processing.

[0006] 2. Algorithm optimization

[0007] Algorithm optimization is an important means to improve data processing efficiency. However, existing algorithms often show certain limitations when dealing with complex data types and changing data environments. For example, traditional data cleaning algorithms may not be able to accurately identify and process all types of erroneous data; data conversion algorithms may be inefficient when processing complex data structures; and data integration algorithms may face challenges in data consistency and integrity when processing multi-source heterogeneous data.

[0008] 3. Data cleaning

[0009] Data cleaning is an indispensable part of the data processing process. Existing data cleaning methods mainly include missing value processing, outlier detection, duplicate value removal, etc. However, these methods may not be able to cope with complex data. For example, for data sets with complex structures (such as nested JSON, XML, etc.), existing cleaning methods may not be able to effectively identify and process erroneous data. In addition, errors in the data cleaning process may propagate to subsequent data processing links, affecting the overall data quality.

[0010] 4. Data Conversion

[0011] Data conversion is the process of converting data from one format or structure to another. Existing data conversion methods usually rely on predefined conversion rules or templates, which may lack flexibility when dealing with complex and changing data. In addition, performance bottlenecks in the data conversion process may also lead to a decrease in overall processing efficiency.

[0012] 5. Data Integration

[0013] Data integration is the process of combining data from different data sources into a unified data set. Existing data integration methods often face challenges in data consistency and integrity. For example, data from different data sources may have differences in format, encoding, semantics, etc. These differences may lead to inconsistencies or omissions in the data during the integration process. In addition, performance issues during the data integration process may also affect the overall data processing efficiency.

[0014] It can be seen that the existing water data processing process still has many requirements for optimization and performance improvement. The present invention aims to solve this technical problem. Summary of the invention

[0015] In this regard, the present invention provides a data processing method, system, electronic equipment, computer storage medium and computer program product for a water management system to solve the above technical problems.

[0016] The present invention discloses a data processing method for a water management system, which is applied to a data middle station of the water management system. The method comprises the following steps:

[0017] S1. Optimize data processing flow: Introduce parallel processing mechanism to split water data processing tasks into multiple subtasks that can be executed in parallel. At the same time, use dynamic scheduling algorithm to dynamically adjust task allocation according to the real-time status of system resources;

[0018] S2. Strengthen data cleaning and conversion capabilities: Use data cleaning tools based on intelligent data cleaning algorithms to automatically identify and process various types of water data errors; use machine learning model conversion engines to optimize water data conversion and integration processes;

[0019] S3. Data integration: Build a unified data integration platform that can automatically identify and parse data formats and structures from different data sources.

[0020] Optionally, step S1 specifically includes:

[0021] S11, analyze the existing data processing flow and identify the bottleneck links and parts that can be parallelized;

[0022] S12, design a parallel processing solution to split the water affairs data processing task into multiple subtasks that can be executed in parallel;

[0023] S13, implementing a dynamic scheduling algorithm to dynamically adjust task allocation according to the real-time status of system resources;

[0024] S14, integrate the optimized water data processing process into the data middle-office system and conduct testing and verification.

[0025] Optionally, the data center uses a distributed computing framework to process water data.

[0026] Optionally, the S12, designing a parallel processing solution to split the water affairs data processing task into multiple subtasks that can be executed in parallel, includes:

[0027] S121, identifying and analyzing water affairs data processing tasks, extracting data volume indicators, data type diversity indicators, overall correlation indicators between data, and data spatiotemporal distribution indicators;

[0028] S122, inputting the data volume index, the data type diversity index, the overall correlation index between the data, and the data spatiotemporal distribution index into a task splitting prediction model based on a deep learning algorithm at the same time, and determining the optimal number of task splits according to the output result of the task splitting prediction model;

[0029] S123, pre-dividing the water affairs data processing task into a plurality of water affairs data processing subtasks according to the task splitting number;

[0030] S124, monitor and obtain the usage of computing resources, and evaluate the overall execution efficiency of each of the water affairs data processing subtasks according to the usage of computing resources. If the overall execution efficiency meets the efficiency index, split the water affairs data processing task into multiple water affairs data processing subtasks; if the overall execution efficiency does not meet the efficiency index, repeat the above steps S122 and S123 until the overall execution efficiency meets the efficiency index.

[0031] Optionally, determining the optimal number of task splits according to the output result of the task split prediction model includes:

[0032] The task splitting prediction model outputs the splitting difficulty coefficient of the water affairs data processing task, and calculates the optimal number of task splits based on the splitting difficulty coefficient.

[0033] Optionally, the water affairs data processing task is pre-divided into a plurality of water affairs data processing subtasks according to the task splitting number, including:

[0034] Determine a task priority value based on the urgency and completion time limit of each parallelizable task included in the water affairs data processing task, and sort the tasks from largest to smallest according to the task priority value;

[0035] The task priority values ​​of each task are clustered and grouped according to the number of task splits to obtain multiple priority intervals, and each task in a priority interval with a higher order is executed first, and each task in each priority interval is set to be processed in the same batch.

[0036] The present invention also discloses a data processing system for a water management system, which is applied to a data middle station of the water management system. The system includes a processor and a memory. The processor calls and executes a computer code in the memory to implement the following steps:

[0037] S1. Optimize data processing flow: Introduce parallel processing mechanism to split water data processing tasks into multiple subtasks that can be executed in parallel. At the same time, use dynamic scheduling algorithm to dynamically adjust task allocation according to the real-time status of system resources;

[0038] S2. Strengthen data cleaning and conversion capabilities: Use data cleaning tools based on intelligent data cleaning algorithms to automatically identify and process various types of water data errors; use machine learning model conversion engines to optimize water data conversion and integration processes;

[0039] S3. Data integration: Build a unified data integration platform that can automatically identify and parse data formats and structures from different data sources.

[0040] The present invention also discloses an electronic device, comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement any of the above methods.

[0041] The present invention also discloses a computer storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement any of the above methods.

[0042] The present invention also discloses a computer program product, wherein the computer program product includes a computer program, and the computer program is executed by a processor of an electronic device to implement any of the above methods.

[0043] The present invention realizes comprehensive optimization of data processing flow and performance improvement. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 It is a flow chart of a data processing method of a water management system disclosed in an embodiment of the present invention;

[0046] Figure 2 It is a structural schematic diagram of a data processing system of a water management system disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following is a description of the implementation of the present application by specific specific embodiments. People familiar with the technology can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0048] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0049] like Figure 1 As shown, an embodiment of the present invention discloses a data processing method for a water management system, which is applied to a data middle station. The method includes the following steps:

[0050] S1. Optimize data processing flow: Introduce parallel processing mechanism to split water data processing tasks into multiple subtasks that can be executed in parallel. At the same time, use dynamic scheduling algorithm to dynamically adjust task allocation according to the real-time status of system resources.

[0051] The present invention first optimizes the data processing process. Traditional data processing processes often use a sequential execution method, resulting in low processing efficiency. The present invention introduces a parallel processing mechanism to split the water data processing task into multiple subtasks that can be executed in parallel. This mechanism can fully utilize the computing power of multi-core processors, significantly improve the data processing speed, and shorten the processing time. At the same time, a dynamic scheduling algorithm is used to dynamically adjust task allocation according to the real-time status of system resources to ensure efficient use of resources. This optimization not only significantly improves the data processing speed, but also enhances the flexibility and scalability of the system.

[0052] S2. Strengthen data cleaning and conversion capabilities: Use data cleaning tools based on intelligent data cleaning algorithms to automatically identify and process various types of water data errors; use machine learning model conversion engines to optimize water data conversion and integration processes.

[0053] For data cleaning, the present invention has developed a data cleaning tool based on intelligent data cleaning algorithms (such as machine learning algorithms, natural language processing NLP, etc.), which can automatically identify and process various types of water service data errors, such as missing values, outliers, duplicate values, etc., and support the parsing and cleaning of complex data structures to ensure the accuracy and comprehensiveness of data cleaning. The data cleaning tool supports the parsing and cleaning of multiple data formats.

[0054] In addition, the present invention also introduces a conversion engine based on a machine learning model, which is responsible for converting water service data from one format or structure to another format or structure, supporting complex data structure conversion and encoding conversion, and ensuring the accuracy and integrity of the data during the conversion process. At the same time, the conversion engine also optimizes the data conversion and integration process, improves the efficiency of water service data conversion, improves the intelligence and adaptability of the algorithm, reduces manual intervention, and reduces the error rate.

[0055] S3. Data integration: Build a unified data integration platform that can automatically identify and parse data formats and structures from different data sources.

[0056] In terms of data integration, the present invention builds a unified data integration platform. The platform supports the integration of multi-source heterogeneous data, can automatically identify and analyze the data format and structure from different data sources, and ensure the consistency and integrity of the data through technical means such as data lineage tracking. At the same time, the platform also provides a wealth of data integration tools and interfaces, which facilitates users to perform customized data integration operations according to actual needs. This unified data integration platform not only improves the efficiency and quality of data integration, but also provides strong support for data sharing and collaborative work within the enterprise.

[0057] The core design concept of the present invention is to optimize the water data processing flow of the data center through a series of innovative technical means, and improve the efficiency, accuracy and resource utilization efficiency of water data processing. Specifically, the present invention aims to solve the problems of low efficiency, insufficient accuracy and high resource consumption faced by existing data processing methods when processing large-scale and highly complex data. By introducing parallel processing mechanisms, strengthening data cleaning and conversion capabilities, and building a unified data integration platform, the present invention achieves comprehensive optimization of the data processing flow and performance improvement.

[0058] The beneficial technical effects are as follows:

[0059] 1) Parallel processing mechanism: The present invention introduces a parallel processing mechanism into the water affairs data processing flow, dividing the large-scale data set into multiple small batches for parallel processing. This mechanism can fully utilize the computing power of multi-core processors, significantly improve data processing speed, and shorten processing time.

[0060] 2) Dynamic Scheduling Algorithm: In order to further optimize resource utilization, the present invention designs a dynamic scheduling algorithm. The algorithm can monitor the load of system resources in real time and dynamically adjust the allocation of processing tasks according to the load. This dynamic adjustment can ensure efficient utilization of resources and avoid resource waste.

[0061] 3) Intelligent data cleaning algorithm: In view of the complexity and diversity of the water service data cleaning process, the present invention adopts an intelligent data cleaning algorithm. The algorithm can automatically identify and process various types of data errors, such as missing values, abnormal values, duplicate values, etc., and support the analysis and cleaning of complex data structures. This intelligence ensures the accuracy and comprehensiveness of data cleaning.

[0062] 4) Machine learning optimization: The present invention introduces a machine learning model in the process of data processing and conversion. The data conversion and integration process is optimized by training the model to improve the intelligence and adaptability of the algorithm. This optimization can reduce manual intervention, reduce the error rate, and further improve the efficiency and quality of data processing.

[0063] 5) Unified data integration platform: In order to solve the problem of multi-source heterogeneous data integration, the present invention builds a unified data integration platform. The platform supports the access of multiple data sources and the conversion of data formats, and ensures the consistency and integrity of data through data lineage tracking technology. This unified platform not only improves the efficiency and quality of data integration, but also provides strong support for data sharing and collaborative work within the enterprise.

[0064] Optionally, step S1 specifically includes:

[0065] S11, analyze the existing data processing flow and identify the bottleneck links and parts that can be parallelized;

[0066] S12, design a parallel processing solution to split the water affairs data processing task into multiple subtasks that can be executed in parallel;

[0067] S13, implementing a dynamic scheduling algorithm to dynamically adjust task allocation according to the real-time status of system resources;

[0068] S14, integrate the optimized water data processing process into the data middle-office system and conduct testing and verification.

[0069] Optionally, the data center uses a distributed computing framework to process water data.

[0070] The data center can use distributed computing frameworks such as Hadoop and Spark to process water data, which can further improve processing efficiency, especially for large-scale data sets.

[0071] Optionally, the S12, designing a parallel processing solution to split the water affairs data processing task into multiple subtasks that can be executed in parallel, includes:

[0072] S121, identifying and analyzing water affairs data processing tasks, extracting data volume indicators, data type diversity indicators, overall correlation indicators between data, and data spatiotemporal distribution indicators;

[0073] S122, inputting the data volume index, the data type diversity index, the overall correlation index between the data, and the data spatiotemporal distribution index into a task splitting prediction model based on a deep learning algorithm at the same time, and determining the optimal number of task splits according to the output result of the task splitting prediction model;

[0074] S123, pre-dividing the water affairs data processing task into a plurality of water affairs data processing subtasks according to the task splitting number;

[0075] S124, monitor and obtain the usage of computing resources, and evaluate the overall execution efficiency of each of the water affairs data processing subtasks according to the usage of computing resources. If the overall execution efficiency meets the efficiency index, split the water affairs data processing task into multiple water affairs data processing subtasks; if the overall execution efficiency does not meet the efficiency index, repeat the above steps S122 and S123 until the overall execution efficiency meets the efficiency index.

[0076] In this embodiment, the present invention is designed to use a task splitting prediction model based on a deep learning algorithm to determine the specific number into which the water affairs data processing task should be split. Specifically, the water affairs data processing task is first identified and analyzed, and the data volume index, data type diversity index, data overall correlation index, and data spatiotemporal distribution index are extracted. Among them, the data volume index refers to the number of data to be processed contained in the water affairs data processing task, and the data volume index is positively correlated with the number of data to be processed; the data type diversity index refers to the number of data types contained in the water affairs data processing task, and the data type diversity index is positively correlated with the number of data types; the overall correlation index between data refers to the weighted mean of the correlation degree values ​​between the various types of data contained in the water affairs data processing task, and the overall correlation index between data is positively correlated with the weighted mean; the data spatiotemporal distribution index value is the maximum time span and maximum spatial span of the data contained in the water affairs data processing task, wherein the spatial span refers to the geographical distance between the data sending terminals, and the overall correlation index between data is positively correlated with the maximum time span and the maximum spatial span.

[0077] Then, the task splitting prediction model based on the deep learning algorithm is used to conduct a comprehensive analysis of the aforementioned data volume index, data type diversity index, overall correlation index between data, and data spatiotemporal distribution index to obtain the corresponding prediction results, and the optimal number of task splits is determined based on the prediction results. At this point, the water affairs data processing task can be pre-split into multiple water affairs data processing subtasks based on the number of task splits obtained. The pre-splitting here refers to simulated splitting rather than real splitting.

[0078] Finally, the usage of computing resources available to the water affairs data center is monitored and obtained, and the overall execution efficiency of the pre-split water affairs data processing subtasks is evaluated based on the usage of computing resources. If the overall execution efficiency meets the efficiency index, the real splitting is performed according to the pre-splitting plan; if the overall execution efficiency does not meet the efficiency index, it indicates that the number of task splits determined above is not optimal, and the above steps S122 and S123 need to be repeated until the overall execution efficiency meets the efficiency index.

[0079] The usage of computing resources refers to the occupancy or remaining status of the CPU, memory, network bandwidth, etc., such as the usage / remaining amount of CPU computing power, the usage / remaining amount of memory, and the usage / remaining amount of network bandwidth.

[0080] Optionally, determining the optimal number of task splits according to the output result of the task split prediction model includes:

[0081] The task splitting prediction model outputs the splitting difficulty coefficient of the water affairs data processing task, and calculates the optimal number of task splits based on the splitting difficulty coefficient.

[0082] In this embodiment, the task splitting prediction model constructed by the present invention is specifically used to analyze the splitting difficulty of water affairs data processing tasks, that is, by comprehensively analyzing the data volume index, data type diversity index, overall correlation index between data, and data spatiotemporal distribution index of water affairs data processing tasks, the corresponding splitting difficulty coefficient is obtained. Then, the optimal number of task splits is calculated based on the splitting difficulty coefficient.

[0083] The number of task splits can be obtained by matching the split difficulty coefficient with preset control data, and the preset control data is shown in the following table:

[0084] Number of task splits Split Difficulty Coefficient 5 0.8 7 0.6 8 0.5

[0085] The above table is only for illustration, and is not intended to limit the preset control data of the present invention to only the above-mentioned situation. Obviously, there is a negative correlation between the number of task splits and the splitting difficulty coefficient, that is, the higher the splitting difficulty coefficient, the larger the amount of data within the water affairs data processing task, the more data types, the higher the correlation between the data, and the more complex the spatial and temporal distribution of the data. At this time, the fewer the number of task splits are set, thereby reducing the probability of data processing errors caused by too small splitting granularity; the lower the splitting difficulty coefficient, the smaller the amount of data within the water affairs data processing task, the fewer data types, the lower the correlation between the data, and the lower the complexity of the spatial and temporal distribution of the data. At this time, the number of task splits is appropriately increased, thereby increasing the number of parallel tasks to improve data processing efficiency.

[0086] Among them, the task splitting prediction model is preferably built based on ResNet (Res i dual Network) and Transformer. The specific structure and training process of the model will not be described here, and the existing method can be used.

[0087] Optionally, the water affairs data processing task is pre-divided into a plurality of water affairs data processing subtasks according to the task splitting number, including:

[0088] Determine a task priority value based on the urgency and completion time limit of each parallelizable task included in the water affairs data processing task, and sort the tasks from largest to smallest according to the task priority value;

[0089] The task priority values ​​of each task are clustered and grouped according to the number of task splits to obtain multiple priority intervals, and each task in a priority interval with a higher order is executed first, and each task in each priority interval is set to be processed in the same batch.

[0090] In this embodiment, the task splitting in the present invention can be to group the tasks that can be parallelized in the original water affairs data processing task, determine the processing order of each group, and set a parallel processing mechanism for each task in the group, that is, execute them together. Specifically, first evaluate the task priority value according to the urgency and completion time limit of each task contained in the water affairs data processing task. The higher the urgency and the shorter the completion time limit, the higher the corresponding task priority value, and the lower the urgency and the longer the completion time limit, the lower the corresponding task priority value. According to the task priority value, sort the tasks from large to small, so that the priority values ​​of each task can be clustered and grouped, and multiple priority intervals are obtained, and the corresponding tasks in each priority interval are divided into a group. When processing, the group at the front of the priority interval is processed first (that is, the minimum value of the task priority value of each task in the group is higher than the maximum value of the task priority value in other groups), and the tasks in the group are synchronously processed according to the parallel processing mechanism; after the tasks of the group are processed, the group corresponding to the next priority interval is processed, and so on, until the tasks in all groups are executed.

[0091] like Figure 2 As shown, an embodiment of the present invention further discloses a data processing system for a water management system, which is applied to a data middle station of the water management system. The system includes a processor and a memory. The processor calls and executes a computer code in the memory to implement the following steps:

[0092] S1. Optimize data processing flow: Introduce parallel processing mechanism to split water data processing tasks into multiple subtasks that can be executed in parallel. At the same time, use dynamic scheduling algorithm to dynamically adjust task allocation according to the real-time status of system resources;

[0093] S2. Strengthen data cleaning and conversion capabilities: Use data cleaning tools based on intelligent data cleaning algorithms to automatically identify and process various types of water data errors; use machine learning model conversion engines to optimize water data conversion and integration processes;

[0094] S3. Data integration: Build a unified data integration platform that can automatically identify and parse data formats and structures from different data sources.

[0095] An embodiment of the present invention further discloses an electronic device, comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the method described in the above embodiment.

[0096] An embodiment of the present invention further discloses a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is executed by a processor to implement the method described in the above embodiment.

[0097] An embodiment of the present invention further discloses a computer program product, wherein the computer program product includes a computer program, and the computer program is executed by a processor of an electronic device to implement any of the above methods.

[0098] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0099] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0100] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0101] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0102] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0103] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-On lyMemory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0104] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A data processing method for a water management system, applied to a data middle station of a water management system, characterized in that: The method comprises the following steps: S1. Optimize data processing flow: Introduce parallel processing mechanism to split water data processing tasks into multiple subtasks that can be executed in parallel. At the same time, use dynamic scheduling algorithm to dynamically adjust task allocation according to the real-time status of system resources; S2. Strengthen data cleaning and conversion capabilities: Use data cleaning tools based on intelligent data cleaning algorithms to automatically identify and process various types of water service data errors; Optimize the water data conversion and integration process using a machine learning-based model conversion engine; S3. Data integration: Build a unified data integration platform that can automatically identify and parse data formats and structures from different data sources.

2. A data processing method for a water management system according to claim 1, characterized in that: Step S1 specifically includes: S11, analyze the existing data processing flow and identify the bottleneck links and parts that can be parallelized; S12, design a parallel processing solution to split the water affairs data processing task into multiple subtasks that can be executed in parallel; S13, implementing a dynamic scheduling algorithm to dynamically adjust task allocation according to the real-time status of system resources; S14, integrate the optimized water data processing process into the data middle-office system and conduct testing and verification.

3. A data processing method for a water management system according to claim 2, characterized in that: The data center uses a distributed computing framework to process water data.

4. A data processing method for a water management system according to claim 3, characterized in that: S12, designing a parallel processing solution to split the water affairs data processing task into multiple subtasks that can be executed in parallel, including: S121, identifying and analyzing water affairs data processing tasks, extracting data volume indicators, data type diversity indicators, overall correlation indicators between data, and data spatiotemporal distribution indicators; S122, inputting the data volume index, the data type diversity index, the overall correlation index between the data, and the data spatiotemporal distribution index into a task splitting prediction model based on a deep learning algorithm at the same time, and determining the optimal number of task splits according to the output result of the task splitting prediction model; S123, pre-dividing the water affairs data processing task into a plurality of water affairs data processing subtasks according to the task splitting number; S124, monitor and obtain the usage of computing resources, and evaluate the overall execution efficiency of each of the water affairs data processing subtasks according to the usage of computing resources. If the overall execution efficiency meets the efficiency index, split the water affairs data processing task into multiple water affairs data processing subtasks; if the overall execution efficiency does not meet the efficiency index, repeat the above steps S122 and S123 until the overall execution efficiency meets the efficiency index.

5. The data processing method of a water management system according to claim 4, characterized in that: Determining the optimal number of task splits according to the output result of the task split prediction model includes: The task splitting prediction model outputs the splitting difficulty coefficient of the water affairs data processing task, and calculates the optimal number of task splits based on the splitting difficulty coefficient.

6. A data processing method for a water management system according to claim 5, characterized in that: The water affairs data processing task is pre-divided into a plurality of water affairs data processing subtasks according to the task splitting number, including: Determine a task priority value based on the urgency and completion time limit of each parallelizable task included in the water affairs data processing task, and sort the tasks from largest to smallest according to the task priority value; The task priority values ​​of each task are clustered and grouped according to the number of task splits to obtain multiple priority intervals, and each task in a priority interval with a higher order is executed first, and each task in each priority interval is set to be processed in the same batch.

7. A data processing system for a water management system, applied to a data middle station of the water management system, the system comprising a processor and a memory, characterized in that: The processor calls and executes the computer code in the memory to implement the following steps: S1. Optimize data processing flow: Introduce parallel processing mechanism to split water data processing tasks into multiple subtasks that can be executed in parallel. At the same time, use dynamic scheduling algorithm to dynamically adjust task allocation according to the real-time status of system resources; S2. Strengthen data cleaning and conversion capabilities: Use data cleaning tools based on intelligent data cleaning algorithms to automatically identify and process various types of water service data errors; Optimize the water data conversion and integration process using a machine learning-based model conversion engine; S3. Data integration: Build a unified data integration platform that can automatically identify and parse data formats and structures from different data sources.

8. An electronic device, comprising: At least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 6.

9. A computer storage medium storing a computer program, characterized in that: The computer program is executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that: The computer program is executed by a processor of an electronic device to implement the method according to any one of claims 1 to 6.

Images