An ai-based water affair data quality intelligent processing method and system

Abstract

The application relates to an AI-based water affair data quality processing method and system, and relates to the field of data quality processing, which comprises the following steps: establishing a data monitoring channel, a data standard inspection rule library and a data hierarchical area, obtaining initial water affair data, writing the initial water affair data into a data buffer pool as a to-be-inspected data stream, extracting target business data items, determining value interval thresholds and format conditions according to the data standard inspection rule library, intercepting the to-be-inspected data stream when the target business data items do not satisfy the format conditions or the value interval thresholds, marking the to-be-inspected data stream that passes the comparison as qualified water affair data, distributing the qualified water affair data to the data hierarchical area, constructing a standard data detail table, extracting business unique identifiers from the standard data detail table, splicing the business unique identifiers, generating a comprehensive water affair theme model, collecting a data calling request, encapsulating the comprehensive water affair theme model into a standard application interface, and outputting the comprehensive water affair theme model. The application has the effect of fully mining the value of data.

Description

Technical Field

[0001] This invention relates to the field of data quality processing, and in particular to an AI-based intelligent processing method and system for water data quality. Background Technology

[0002] Data quality processing is the process by which enterprises improve the accuracy and integrity of data through cleaning, deduplication, standardization, and other means. Each department often builds its own vertical business system.

[0003] Existing water data quality processing typically employs non-real-time techniques during data collection and quality inspection, frequently performing full or periodic batch extractions of the source business system database during off-peak hours. Before data is implemented, it relies on manual pre-setting of fixed rules for periodic post-event checks. During data integration and sharing, data is transmitted by opening database account passwords or physical files.

[0004] The existing water data quality processing directly and frequently reads the source business system database, which is not only inefficient but also causes the source system to lag. Secondly, the periodic checks and rigid rules after the fact prevent dirty data from being blocked during the process, which can lead to misjudgment of data during peak business periods and cause valuable data to be buried. In addition, the independent construction of vertical business systems by each department leads to a serious waste of enterprise computing and storage resources, and the value of data is difficult to be fully explored. Summary of the Invention

[0005] To fully unlock the value of data, this invention provides an AI-based method and system for water data quality processing.

[0006] In a first aspect, the present invention provides an AI-based method for water data quality processing, employing the following technical solution: An AI-based water data quality processing method includes: Establish a data monitoring channel, a data standard verification rule base, and a data tiered area consisting of multiple storage levels. Obtain initial water affairs data according to the data monitoring channel and write it into the data buffer pool as a data stream to be inspected. Extract target business data items from the data stream to be inspected; Determine the numerical range thresholds and format conditions based on the data standard verification rule base; When any target business data item does not meet its corresponding format conditions or numerical range threshold, the corresponding data stream to be inspected is intercepted, and all data streams to be inspected that pass the comparison are marked as qualified water affairs data. Qualified water affairs data are mapped and allocated to data hierarchical areas, and corresponding standard data detail tables are constructed. Extract the business unique identifier from the corresponding standard data detail table, and concatenate the standard data detail tables with the same business unique identifier across different standard data detail tables to generate a comprehensive water affairs theme model. Data collection request; Based on the data retrieval request, the integrated water affairs theme model is encapsulated into a standard application interface and output.

[0007] By adopting the above technical solutions, water affairs data from the source business system can be uniformly collected using a data monitoring channel, and the initial water affairs data can be buffered through a data buffer pool to reduce the processing pressure caused by instantaneous data writing. At the same time, the target business data items in the data stream to be inspected are formatted and numerically validated using a data standard verification rule base, intercepting abnormal data and forming a standard data detail table for qualified data, which can improve the data quality entering the subsequent data model. Furthermore, different standard data detail tables are concatenated across tables using a unique business identifier to generate a comprehensive water affairs theme model, and this comprehensive water affairs theme model is encapsulated as a standard application interface for output, thereby fully exploring the value of the data.

[0008] Optionally, initial water data may be obtained from the data monitoring channel, including: Obtain underlying business data; Establish a connection between the data monitoring channel and the underlying business data, and obtain the change log file in the underlying business data; Extract the operation content recorded in the change log file, and extract the corresponding timestamp and business source identifier based on the operation content; The timestamp, business source identifier, and operation content are encapsulated as initial water affairs data.

[0009] By adopting the above technical solution, the data monitoring channel can establish a connection with the underlying business data and extract operation content, timestamps, and business source identifiers from the change log file, which are then uniformly packaged into initial water affairs data. This allows for the acquisition of data change information such as additions, updates, and deletions without directly modifying the data tables within the source business system, improving the continuity and traceability of data collection.

[0010] Optionally, writing to the data buffer pool includes: The data buffer pool is configured as a multi-node distributed cluster; Parse the initial water data, extract the timestamp and business source identifier, and concatenate the timestamp and business source identifier as the routing key; A preset routing distribution algorithm is executed on the routing key to calculate and determine the corresponding target node partition index; Based on the target node partition index, the initial water data is distributed and written to the corresponding multi-node distributed cluster in the data buffer pool for queuing and buffering.

[0011] By adopting the above technical solution, when initial water data is written to the data buffer pool, a routing key can be generated based on the timestamp and business source identifier, and then written to the corresponding node partition through a routing distribution algorithm. This allows a large amount of instantaneously written data to be distributed across different nodes for queuing and buffering, reducing the data write pressure on a single node and improving the stability and processing efficiency of data access.

[0012] Optionally, all data streams that pass the comparison and are marked as qualified water data include: Extract blank rules and format matching rules for the target business data item from the data standard verification rule library as format conditions, and extract the historical normal operation data of the target business data item; The threshold value range is determined based on historical normal operation data; Extract the data content format and data values ​​based on the target business data items; When the data content format does not meet the format conditions, or the data value exceeds the value range threshold, an interception command is generated for the data stream to be inspected, and the data stream is intercepted. When the data content format meets the format conditions and the data value is within the threshold of the value range, the data stream to be inspected is marked as qualified water affairs data.

[0013] By adopting the above technical solution, blank rules, format matching rules, and historical normal operation data can be extracted from the data standard verification rule base, and numerical range thresholds can be calculated based on the historical normal operation data. This not only allows for the determination of whether target business data items contain null values ​​or have format errors, but also enables the determination of whether current values ​​are abnormal by combining historical normal data, thereby improving the accuracy of abnormal data identification.

[0014] Optionally, after interception, the following can be included: Define the corresponding target business data item as abnormal data and write it to the preset abnormal data station in real time; Analyze the abnormal data in the abnormal data station to extract the error causes and data sources; The cause of the error is fed back from the data source to obtain correction information, and the correction information is then used as the target business data item again. The number of corrections is obtained based on the correction information; Determine the allowed number of times based on the blank rule; If the number of corrections exceeds the target allowed number, the corresponding abnormal data is obtained, treated as red line data, and output.

[0015] By adopting the above technical solution, intercepted data can be written to the abnormal data station, and the cause of the error and the data source can be extracted from the abnormal data. Then, the error is fed back to the corresponding data source to obtain correction information. This forms a closed loop for abnormal data discovery, cause location, and feedback correction, facilitating continuous tracking and correction of abnormal data.

[0016] Optionally, qualified water data can be mapped and assigned to construct standard data detail tables and generate integrated water subject models, specifically including: In the pre-defined storage database, the technology buffer layer, near-source model layer, integration model layer and application mart layer are divided and isolated from each other according to the technology buffer, near-source model, integration model and application mart stages of data flow. Qualified water data is input into the technical buffer layer, and then distributed and loaded into the near-source model layer by the technical buffer layer, and converted into a standard data detail table; Based on the unique business identifier, the standard data detail tables of different business systems are spliced ​​across tables in the integration model layer to generate a comprehensive water affairs theme model, which is then solidified and loaded into the application mart layer.

[0017] By adopting the above technical solution, the data flow stage is divided into technical buffer, near-source model, integration model and application mart, and independent storage layers are formed in the storage database. This allows data access, standardization processing, theme integration and application output to be completed in different layers, reducing data mixing between different processing stages.

[0018] Optionally, cross-table concatenation of standard data detail tables belonging to different business systems includes: Extract the original subsystem numbers from the standard data detail tables belonging to different business systems; The unified primary key of the middle platform is calculated and generated based on the current system time and server node number, and a global code lookup table is constructed. The original numbers of each subsystem are extracted and entered into the global code lookup table for retrieval and matching. The original numbers of each subsystem are then uniformly replaced with the corresponding unified primary key of the middle platform. Update the corresponding standard data detail table with the same unified primary key of the middle platform as the corresponding business unique identifier; For standard data detail tables belonging to different business systems, but having the same unique business identifier as the basis for cross-table concatenation, perform horizontal concatenation operations.

[0019] By adopting the above technical solution, the original subsystem IDs can be extracted from the standard data detail tables of different business systems, and a conversion correspondence can be established through a unified primary key and global code lookup table in the middle platform. This allows the original IDs from different source business systems to be uniformly converted into primary key identifiers recognizable by the middle platform, fully leveraging the value of the data.

[0020] Optionally, after generating the integrated water affairs thematic model, it also includes: Historical and current data are extracted based on a comprehensive water affairs thematic model; Determine the dynamic tolerance trend range using historical data; The current offset is determined by combining historical and current data. If the current offset is within the dynamic tolerance trend range, jump to execute the data collection request; If the current offset is not within the dynamic tolerance trend range, obtain the association path between the fields of each table during the horizontal splicing process of the standard data detail table; Based on the associated path, determine all corresponding associated nodes, and combine the associated nodes and associated path to determine the splicing node; The data association graph is determined by combining splicing nodes and associated paths; Based on the data association graph, the corresponding current data is traced back to identify abnormal targets, and the results are output.

[0021] By adopting the above technical solution, historical and current data can be extracted based on a comprehensive water affairs thematic model. A dynamic tolerance trend range can be determined based on the historical data, and the current offset can be calculated by combining this with the current data. This allows for the assessment of whether current data exhibits abnormal fluctuations based on historical changes, improving the ability to identify trend-related anomalies.

[0022] Optionally, the integrated water management model can be encapsulated into a standard application interface and output, specifically including: Receive data retrieval requests and extract the security access token from the data retrieval requests; The security access token is matched and verified against the preset permission list to determine the access permissions of the interface. When the interface access permission is granted, the access level corresponding to the data call request is obtained based on the interface access permission. Based on the access level, retrieve the viewable content corresponding to the access level in the integrated water affairs theme model as the access content; The accessed content is encapsulated and output through standard application interfaces; When the interface access permission is no, the application reason is obtained from the data call request, the application reason is approved, and the approval result is obtained. If the approval result is passed, a temporary access token, temporary permissions, and temporary access level are generated based on the approval result. The corresponding access content is then filtered based on the temporary access level and output through the standard application interface. If the approval result is negative, no access content will be output, and an access denial message will be generated.

[0023] By adopting the above technical solution, before outputting the integrated water affairs theme model, the security access token in the data call request is extracted and matched with the permission roster to determine whether the requester has the interface access permission, thereby reducing the risk of unauthorized data call.

[0024] Secondly, this invention provides an AI-based water data quality processing system, employing the following technical solution: An AI-based water data quality processing system includes a data acquisition module for acquiring data retrieval requests. The memory stores a program for implementing an AI-based water data quality processing control method as described above; The processor loads and executes programs stored in memory.

[0025] In summary, the present invention has at least one of the following beneficial technical effects: 1. Initial water data is acquired through a data monitoring channel and then buffered, quality inspected, layered table creation, cross-table splicing, and interface output are performed sequentially. This enables water data from different source business systems to be uniformly accessed, verified, integrated, and accessed, thereby fully exploring the value of the data. 2. By concatenating timestamps and business source identifiers to generate routing keys, and writing initial water data into the corresponding node partition in the data buffer pool according to the routing keys, the processing pressure during the instantaneous writing of massive amounts of data can be distributed, thereby improving the stability of water data access and buffer processing; 3. By verifying data access requests through secure access tokens and permission lists, and outputting access content within the corresponding permission scope according to the access level, the risk of unauthorized access to data can be reduced, thereby improving the data security when the integrated water affairs theme model is output externally. Attached Figure Description

[0026] Figure 1 This is a flowchart of an AI-based water data quality processing method.

[0027] Figure 2 This is a flowchart for obtaining initial water data based on the data monitoring channel. Detailed Implementation

[0028] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0029] An AI-based water data quality processing method establishes a data monitoring channel in the corresponding project system of the source business system, and builds a data standard verification rule base and a data layering area including a technology buffer layer, a near-source model layer, an integration model layer, and an application mart layer. Data acquired through the data monitoring channel is used as initial water data, written into a data buffer pool, and forms a data stream awaiting quality inspection. Target business data items are extracted from the data stream awaiting quality inspection and filtered using the data standard verification rule base. Data streams that pass the filter are marked as qualified water data. Subsequently, the qualified water data is input into the data layering area to generate a standard data detail table. The standard data detail tables from different business systems are then horizontally concatenated based on unique business identifiers to generate a comprehensive water theme model. Upon receiving a data access request, permission verification is performed on the request. After successful permission verification, the corresponding data in the comprehensive water theme model is standardized, encapsulated, and output, thereby fully leveraging the value of the data.

[0030] Reference Figure 1 This invention discloses an AI-based water data quality processing method, which includes: S100: Establish a data monitoring channel, a data standard inspection rule base, and a data tiered area consisting of multiple storage levels. Obtain initial water affairs data according to the data monitoring channel and write it into the data buffer pool as a data stream to be inspected.

[0031] A data monitoring channel refers to a channel used for bypass reading and parsing of change log files in the source business system database. The data monitoring channel does not perform operations on the business tables of the source business system within the source business system database. The data monitoring channel includes a log reading module for reading change log files according to the database transaction commit order, a log parsing module for parsing change records in the change log files, a position recording module for recording the file number, offset, or transaction sequence number of the read logs, and a message encapsulation module for encapsulating the parsed results into initial water affairs data.

[0032] The data standard verification rule base refers to a rule library composed of rules used to verify the completeness, accuracy, consistency, uniqueness, and validity of water affairs data. In this embodiment, the data standard verification rule base includes verification rules such as rule number, applicable business source identifier, applicable source table name, applicable field name, target business data item, field type, whether nullability is allowed, continuous collection interval, format matching expression, field length range, static upper limit of value, static lower limit of value, threshold generation method, number of historical samples, historical time window, anomaly level, allowed correction times, and activation status. Whether nullability is allowed, continuous collection interval, and missing data judgment conditions are used to form blank rules. Format matching expression, field type, and field length range are used to form format matching rules. Static upper limit of value, static lower limit of value, threshold generation method, and number of historical samples are used to form numerical range rules. The verification rules in the data standard verification rule base are associated with the target business data items in the business data item configuration table through the quality inspection rule number.

[0033] The data tiering area refers to a storage area used to organize and manage water affairs data in layers according to data processing stages, data structures, and data usage scenarios. Initial water affairs data refers to structured water affairs data parsed and encapsulated from the change log files of the source business system database by the data monitoring channel, including at least a timestamp, business source identifier, operation type, source table name, primary key field, changed field name, field value before change, and / or field value after change. The data buffer pool is a distributed cache area used to queue and buffer the instantaneous influx of initial water affairs data. The data stream awaiting quality inspection refers to the initial water affairs data stream that has been written to the data buffer pool but has not yet completed data quality verification.

[0034] By establishing a data monitoring channel, the change log files in the source business system database are bypassed for reading and parsing to obtain initial water affairs data; the initial water affairs data is written into a data buffer pool for queuing and buffering, and the initial water affairs data written into the data buffer pool is used as the data stream to be inspected.

[0035] In this embodiment, a business data item configuration table is pre-established. The business data item configuration table is used to record the correspondence between business source identifier, source table name, source field name, target business data item, field type, data unit, whether the field is mandatory for inspection, and quality inspection rule number.

[0036] The business source identifier identifies the source business system to which the target business data item belongs. The source table name identifies the data table in the source business system where the target business data item resides. The source field name identifies the field name in the source business system where the target business data item resides. The target business data item identifies the standardized field used for data quality inspection. The field type identifies whether the target business data item is numeric, character, date, or enumerated data. Whether the field is mandatory for inspection indicates whether the target business data item must participate in data quality inspection. The quality inspection rule number is used to associate the corresponding verification rule in the data standard inspection rule base.

[0037] S101: Extract target business data items from the data stream to be inspected.

[0038] The target business data item refers to the business field data that needs to be inspected and identified from the data stream to be inspected according to the business data item configuration table.

[0039] By parsing the data stream to be inspected, the business source identifier, source table name, changed field name, changed field value, and primary key field are extracted. The business source identifier, source table name, and changed field name are then input into the business data item configuration table for matching, obtaining the corresponding target business data item, field type, data unit, field mandatory inspection status, and quality inspection rule number. When a configuration record corresponding to the business source identifier, source table name, and changed field name exists in the business data item configuration table, the changed field value corresponding to that changed field name is determined as the target business data item. When multiple changed field names exist in the same data stream to be inspected, they are matched against the business data item configuration table separately, and the field that matches successfully and is marked as mandatory is determined as the target business data item.

[0040] S102: Determine the threshold values ​​and format conditions for the numerical range based on the data standard verification rule library.

[0041] Numerical range thresholds refer to the upper and lower limits of the allowed value range for the target business data item. Format conditions refer to the data format constraints that the target business data item must meet, including field type, character length, format matching expression, or encoding format requirements.

[0042] Based on the quality inspection rule number corresponding to the target business data item, the corresponding verification rule is retrieved from the data standard inspection rule library. The numerical range threshold is determined based on the numerical range rule in the matched verification rule, and the format conditions are determined based on the format matching rule and blank rule in the verification rule.

[0043] S103: When any target business data item does not meet its corresponding format conditions or numerical range threshold, the corresponding data stream to be inspected is intercepted, and all data streams that pass the comparison are marked as qualified water data.

[0044] Qualified water affairs data refers to data in which all mandatory target business data items in the same data stream to be inspected meet the corresponding verification rules.

[0045] If any mandatory target business data item in the same data stream to be inspected fails the blank rule, format matching rule, or fails the numerical range threshold validation when the target business data item is a numeric field, the entire data stream to be inspected will be intercepted. If all mandatory target business data items in the same data stream to be inspected pass the corresponding validation, the data stream to be inspected will be marked as qualified water affairs data.

[0046] S104: Map qualified water data to the data hierarchy area and construct the corresponding standard data detail table.

[0047] A standard data detail table refers to a table in which each independent data record of the acquired qualified water data is standardized according to the indicators, and each record is strictly correlated with a single data collection action or a single business processing flow triggered in the original business system.

[0048] Qualified water affairs data is mapped and assigned to the corresponding partitions of the data hierarchy area. Under the premise of strictly maintaining complete consistency with the original system data granularity, the data is initially standardized in format. Then, these standardized independent detailed entries are listed and stored to form a standard data detail table of relational tables.

[0049] S105: Extract the business unique identifier from the corresponding standard data detail table, and perform cross-table concatenation of standard data detail tables with the same business unique identifier from different standard data detail tables to generate a comprehensive water affairs theme model.

[0050] A business-unique identifier is a globally unique identifier used to uniquely identify the same business object across different business systems. Business objects include users, equipment, pipelines, work orders, billing records, or other water-related business objects. The business-unique identifier serves as a logical join field for cross-table joins and data fusion between different standard data detail tables.

[0051] A comprehensive water affairs theme model refers to a theme data model formed by linking and integrating standard data detail tables from different business systems according to business themes, based on unique business identifiers. The comprehensive water affairs theme model can be represented as a wide theme table or a collection of theme data tables, providing unified data query, statistical analysis, or API call support for various business domains.

[0052] By extracting unique business identifiers from standard data detail tables, data records belonging to different business systems but possessing the same unique business identifier are identified as data records corresponding to the same business object. Using the unique business identifier as the basis for cross-table concatenation, fields from different standard data detail tables are horizontally concatenated and thematically integrated to generate a comprehensive water affairs thematic model.

[0053] S106: Data collection request.

[0054] A data retrieval request refers to information from an external system requesting the retrieval of specified data.

[0055] The data retrieval request is obtained by receiving information from an external system requesting specific data. S107: Based on the data retrieval request, encapsulate the integrated water affairs theme model into a standard application interface and output it.

[0056] The standard application interface refers to the standardized format used to encapsulate and output the data from the integrated water affairs theme model, serving as the sole compliant entry point for external systems to obtain data.

[0057] By requesting the data to be retrieved, the corresponding data in the integrated water affairs theme model is encapsulated in a standardized JSON format, and the encapsulated data is output through a standard application interface.

[0058] Reference Figure 2 Initial water data obtained from the data monitoring channel includes: S200: Obtain underlying business data.

[0059] Underlying business data refers to the business data of the original business system.

[0060] By obtaining the business data from the corresponding project's system, we can use it as the underlying business data.

[0061] S201: Establish a connection between the data monitoring channel and the underlying business data, and obtain the change log file in the underlying business data.

[0062] A change log file is a log file in the source business system database used to record data change operations such as adding, updating, and deleting underlying business data.

[0063] A read-only connection is established between the data monitoring channel and the source business system database corresponding to the underlying business data, and the change log file in the source business system database is read using the database change data capture method.

[0064] S202: Extract the operation content recorded in the change log file, and extract the corresponding timestamp and business source identifier based on the operation content.

[0065] Operation content refers to the operation type and associated data content recorded in the change log file when the underlying business data changes. This includes the operation type, source table name, primary key field, changed field name, field value before change, field value after change, and operation timestamp. Operation types include add, update, and delete. The timestamp refers to the time point when the underlying business data change operation occurred. The business source identifier is a data tag representing the source business system, business module, source database, or source data table to which the changed data belongs.

[0066] By parsing the change records in the change log file, the operation type, source table name, primary key field, changed field name, previous field value, and new field value are extracted as operation content. The corresponding operation timestamp is extracted from the operation content as a timestamp, and the data tag indicating the source business system, business module, source database, or source table to which the changed data belongs is extracted as a business source identifier. When the operation type is "add," the operation content must at least include the primary key field of the added record and the new field value; when the operation type is "update," the operation content must at least include the previous field value and the new field value; when the operation type is "delete," the operation content must at least include the primary key field of the deleted record, the deletion timestamp, and the business source identifier.

[0067] S203: Encapsulate the timestamp, business source identifier, and operation content as initial water data.

[0068] The message encapsulation module encapsulates the timestamp, business source identifier, operation type, source table name, primary key field, changed field name, field value before change, and field value after change according to a preset message format to generate initial water affairs data. The preset message format includes JSON, XML, Protobuf, or other structured message formats.

[0069] Writing to the data buffer pool includes: S300: The data buffer pool is configured as a multi-node distributed cluster.

[0070] A multi-node distributed cluster refers to a server cluster consisting of multiple servers.

[0071] The data buffer pool is configured as a cluster of multiple servers, which is a multi-node distributed cluster.

[0072] S301: Parse the initial water data, extract the timestamp and business source identifier, and concatenate the timestamp and business source identifier as the routing key.

[0073] The routing key is a data identifier used to determine the target node partition of the initial water data in the data buffer pool.

[0074] By obtaining the timestamp and business source identifier carried by the initial water data, the business source identifier is used as a prefix, the timestamp is converted as a suffix, and the data is concatenated to generate navigation coordinates used to allocate to the corresponding server as a routing key.

[0075] S302: Execute the preset route distribution algorithm on the routing key to calculate and determine the corresponding target node partition index.

[0076] The preset routing algorithm refers to the algorithm that determines the corresponding target node partition index based on the routing key. The target node partition index is an identifier used among multiple node partitions in the data buffer pool to determine which node partition the initial water affairs data should be written to.

[0077] The preset routing algorithm can be a hash routing algorithm. A hash routing algorithm takes a routing key as input, performs a hash calculation on the routing key using a preset hash function, obtains the corresponding hash value, and then determines the target node partition index based on the hash value and the number of node partitions in the data buffer pool. Preset hash functions include, but are not limited to, CRC32 hash function, MurmurHash hash function, or other functions that can map a routing key to a numerical hash value.

[0078] S303: Based on the target node partition index, the initial water data is distributed and written to the corresponding multi-node distributed cluster in the data buffer pool for queuing and buffering.

[0079] Based on the target node partition index, determine the corresponding target node partition in the multi-node distributed cluster of the data buffer pool, and write the initial water data into the cache queue corresponding to the target node partition for queuing and buffering.

[0080] Intercepting the corresponding data streams to be inspected, and marking all data streams that pass the comparison as qualified water data includes: S400: Extract blank rules and format matching rules for the target business data item from the data standard verification rule base as format conditions, and extract the historical normal operation data of the target business data item.

[0081] Blank rules are rules used to determine whether a target business data item contains null values, missing values, or gaps in continuous data collection. Format matching rules are rules used to determine whether the data format of a target business data item conforms to pre-defined format requirements. Historical normal operation data refers to historical data corresponding to target business data items that have passed data quality verification and been determined to be normal within a pre-defined sample size.

[0082] Based on the quality inspection rule number corresponding to the target business data item, the corresponding blank rules and format matching rules are retrieved from the data standard inspection rule base, and the matched blank rules and format matching rules are used as format conditions. At the same time, based on the business source identifier, field identifier, and business object identifier of the target business data item, data that meets the preset historical time range or preset sample number is extracted from the historical record database or comprehensive water affairs theme model that has been marked as qualified water affairs data, and is used as historical normal operation data.

[0083] S401: Calculate and determine the threshold value range based on historical normal operation data.

[0084] Retrieve the N most recent historical normal operation data points corresponding to the target business data item, calculate the average μ and standard deviation σ of these N historical normal operation data points, and use μ-kσ as the lower limit of the numerical interval threshold and μ+kσ as the upper limit of the numerical interval threshold. Here, N is the preset sample size, and k is the preset fluctuation coefficient. When the number of historical normal operation data points is less than N, the pre-configured static upper and lower limits for the target business data item in the data standard verification rule library are used as the numerical interval threshold.

[0085] S402: Extract the data content format and data value based on the target business data item.

[0086] Data content format refers to the data representation of field values ​​in the target business data item, including field type, character length, character composition, date format, and encoding format. Data value refers to the numeric field value in the target business data item that can be used to determine numerical ranges.

[0087] By parsing the target business data item, its field type, character length, character composition, date format, or encoding format is extracted as the data content format. When the target business data item is a numeric field, or can be converted to a numeric field, its corresponding value is extracted as the data value. For non-numeric target business data items, blank rules and format matching rules are performed for validation, but numeric range threshold validation is not performed.

[0088] S403: When the data content format does not meet the format conditions, or the data value exceeds the value range threshold, an interception command is generated for the data stream to be inspected, and the interception is performed.

[0089] Interception commands are control commands used to prevent abnormal data streams awaiting quality inspection from entering standard data detail tables, integrated water affairs thematic models, or subsequent data retrieval processes.

[0090] When the data content format does not meet the format conditions, or the data value exceeds the value range threshold, it indicates that the current data stream to be inspected is abnormal. An interception instruction is generated for the data stream to be inspected, and interception is performed based on the interception instruction.

[0091] S404: When the data content format meets the format conditions and the data value is within the threshold of the value range, the data stream to be inspected is marked as qualified water data.

[0092] When all mandatory target business data items in the same data stream to be inspected meet the corresponding blanking rules and format matching rules, and all numerical target business data items are within the corresponding numerical range thresholds, the data stream to be inspected is considered to have passed the data quality verification and is marked as qualified water data. For non-numerical target business data items, meeting the blanking rules and format matching rules is considered as passing the corresponding data quality verification.

[0093] In this embodiment, the data quality anomaly scoring model is trained based on historical qualified water data, historical abnormal data, red line data, and manual review results. The model input includes blank hit characteristics, format matching characteristics, numerical offset characteristics, historical fluctuation characteristics, business source identifier characteristics, and field type characteristics of the target business data item. The model output is an anomaly score. When the anomaly score is greater than or equal to a preset anomaly score threshold, an anomaly risk warning is generated, and the anomaly score is associated with the corresponding target business data item and stored in the anomaly data station or data quality report. This data quality anomaly scoring model is used to provide auxiliary prompts for rule verification results without changing the main process of performing data quality verification based on blank rules, format matching rules, and numerical range thresholds.

[0094] After interception, the following is included: S500: Define the corresponding target business data item as abnormal data and write it to the preset abnormal data station in real time.

[0095] Abnormal data refers to target business data items that fail format condition validation or numerical range threshold validation and are therefore blocked. A pre-configured abnormal data station is a data storage area that is manually configured to centrally store abnormal data and its associated information.

[0096] When the data stream to be inspected is intercepted, the target business data items in the data stream that fail the verification are marked as abnormal data, and the abnormal data and its corresponding data stream identifier, interception time, business source identifier and interception reason are written to the abnormal data station in real time.

[0097] S501: Parse the abnormal data in the abnormal data station and extract the error cause and data source.

[0098] The error reason refers to why the abnormal data failed the corresponding validation rules. The data source refers to the original business system, business module, data table, field, or underlying data acquisition device corresponding to the abnormal data.

[0099] The system parses the abnormal data and its associated information stored in the abnormal data station, extracts the reasons why the abnormal data failed the corresponding validation rules, and identifies the data source corresponding to the abnormal data based on the business source identifier, data table identifier, field identifier, or device identifier carried by the abnormal data.

[0100] S502: Feedback the cause of the error through the data source to obtain correction information, and re-use the correction information as the target business data item.

[0101] Correction information refers to the data content that has been modified, supplemented, or re-collected and resubmitted after the abnormal data has been corrected.

[0102] Based on the data source, the abnormal data, carrying the reason for the error, is fed back to the corresponding source business system, business module, or data acquisition device. The corresponding source business system, business module, data acquisition device, or operator then modifies, completes, or re-acquires the abnormal data, and the resubmitted data content is used as correction information. This correction information is then reinstated as the target business data item.

[0103] S503: Obtain the corresponding number of corrections based on the correction information.

[0104] The number of corrections refers to the cumulative number of times the same abnormal data has been modified, supplemented, or resubmitted after feedback.

[0105] Based on the abnormal data corresponding to the correction information, query the associated correction records and count the cumulative number of resubmissions of the abnormal data as the correction count.

[0106] S504: Determine the number of times the target is allowed based on blank rules.

[0107] The target allowed number of times refers to the maximum number of times the same abnormal data is allowed to be corrected and resubmitted.

[0108] The blanking rules are pre-configured with the allowed number of corrections corresponding to the blanking exception type. When abnormal data triggers a blanking rule, the allowed number of corrections corresponding to the blanking rule is obtained and used as the target allowed number of corrections.

[0109] S505: If the number of corrections exceeds the target allowed number, obtain the corresponding abnormal data, treat it as red line data, and output it.

[0110] Redline data refers to abnormal data that fails the data quality check even after the number of corrections exceeds the target allowable number.

[0111] When the number of corrections exceeds the target allowed number, it indicates that the abnormal data has not been effectively corrected within the allowed correction range. The abnormal data, its corresponding error cause, data source, and correction record are then retrieved and marked as red-line data. Subsequently, the red-line data, along with its corresponding error cause, data source, and number of corrections, are output.

[0112] The qualified water data is mapped and allocated to construct a standard data detail table and generate a comprehensive water theme model, specifically including: S601: In the preset storage database, the technology buffer layer, near-source model layer, integration model layer and application mart layer are divided and isolated into independent layers according to the technology buffer, near-source model, integration model and application mart layers in the data flow stage.

[0113] The pre-configured storage database refers to a data warehouse or database storage space pre-configured for storing, organizing, and managing water affairs data. The technology buffer layer is a storage layer within the storage database used to receive and load data from source business systems. The near-source model layer is a storage layer within the storage database used to maintain the original appearance of the business system data. The integration model layer is a storage layer within the storage database used to integrate and standardize data from different source business systems according to business themes, forming thematic data. The application mart layer is a service layer within the storage database that provides query, analysis, or data retrieval services for specific business application needs.

[0114] In the storage database, the data flow stages are divided into a technology buffer layer, a near-source model layer, an integration model layer, and an application mart layer, so that the data at different data flow stages are logically isolated from each other and loaded, stored, integrated, and output according to the data flow order.

[0115] S602: Input qualified water data into the technical buffer layer, and then distribute and load it into the near-source model layer, converting it into a standard data detail table.

[0116] The qualified water affairs data is input into the technical buffer layer, which then distributes and loads the qualified water affairs data into the near-source model layer according to the data source, data table identifier, or business type. In the near-source model layer, the qualified water affairs data is mapped to fields, formatted uniformly, and type converted. A standard data detail table is generated in a manner that one business record corresponds to one detail record.

[0117] S603: Based on the unique business identifier, standard data detail tables from different business systems are spliced ​​across tables in the integration model layer to generate a comprehensive water affairs theme model, which is then loaded into the application marketplace layer.

[0118] Business-specific identifiers are extracted from standard data detail tables corresponding to different business systems. Standard data detail tables with the same business-specific identifier are identified as data tables belonging to the same business object. In the integration model layer, using the business-specific identifier as the basis for cross-table concatenation, fields in different standard data detail tables are horizontally concatenated and thematically integrated to generate a comprehensive water affairs theme model. Subsequently, the comprehensive water affairs theme model is used as a data service object for business applications and is permanently loaded into the application mart layer for subsequent data querying, analysis, sharing, or interface calls.

[0119] Cross-table concatenation of standard data detail tables belonging to different business systems includes: S700: Extract the original subsystem number from each record in the standard data detail table belonging to different business systems.

[0120] The original number of the subsystem refers to the original number used to identify business objects in the source business system.

[0121] From the standard data detail tables belonging to different business systems, extract the original source business system number of each record, and combine it with the corresponding business source identifier to determine the original identifier of the business object in the source business system in each standard data detail table and use it as the original subsystem number.

[0122] S701: Calculate and generate a unified primary key for the middle platform based on the current system time and server node number, and build a global code lookup table.

[0123] The current system time refers to the system time at the time the unified primary key for the middle platform is to be generated. The server node number refers to the server node identifier that performs the unified primary key generation operation. The unified primary key is a globally unique primary key identifier generated by the system for a business object. The global code lookup table is a data table used to record the conversion and correspondence between business source identifiers, original subsystem numbers, and the unified primary key for the middle platform.

[0124] The system time is converted to a timestamp field corresponding to the system time, and the server node number is converted to a node field. When multiple unified primary keys for the middle platform are generated by the same server node within the same timestamp, an incrementing sequence number field is configured according to the generation order. The timestamp field, server node number field, and incrementing sequence number field are concatenated to generate the unified primary key for the middle platform. The business source identifier, the original subsystem number, and the unified primary key for the middle platform are written as a set of mapping records into a global code lookup table to construct the global code lookup table. The timestamp field is used to represent the generation time of the unified primary key for the middle platform, the server node number field is used to distinguish the server node that generated the unified primary key for the middle platform, and the incrementing sequence number field is used to distinguish multiple primary keys generated by the same server node within the same timestamp.

[0125] S702: Input the original numbers of each subsystem into the global code lookup table for retrieval and matching, and replace the original numbers of each subsystem with the corresponding unified primary key of the middle platform.

[0126] Search matching refers to querying the global code lookup table to see if there is a corresponding mapping record based on the original subsystem number and the corresponding business source identifier.

[0127] The extracted original subsystem IDs are input into the global ID lookup table for retrieval and matching. If a corresponding mapping record is found, the unified primary key of the platform in that mapping record is obtained, and the original subsystem ID in the standard data detail table is replaced with that unified primary key. If there are multiple mapping records in the global ID lookup table where the original subsystem IDs from different business systems correspond to the same unified primary key, then that same unified primary key is used to replace or associate the corresponding records in the standard data detail table.

[0128] S703: Update the corresponding standard data detail table with the same unified primary key of the middle platform as the corresponding business unique identifier.

[0129] When the original subsystem numbers in different standard data detail tables are matched with the same unified primary key of the middle platform through the global code lookup table, the same unified primary key of the middle platform is used as the business unique identifier and written or updated to the corresponding standard data detail table.

[0130] The same unified primary key in the middle platform means that the conversion correspondence recorded in the global code lookup table is consistent, rather than directly determining whether data from different business systems belong to the same business object based solely on the current system time and server node number.

[0131] S704: Perform a horizontal concatenation operation on standard data detail tables belonging to different business systems, but with the same unique business identifier as the basis for cross-table concatenation.

[0132] Horizontal splicing refers to the operation of associating and merging data belonging to the same business object in different standard data detail tables by using a unique business identifier as the association.

[0133] Read the standard data detail tables belonging to different business systems, filter out data records with the same unique business identifier, and use the unique business identifier as the connection condition to perform a horizontal concatenation operation on the fields related to the business object in each standard data detail table to form a comprehensive data record containing field information from multiple business systems.

[0134] After generating the integrated water affairs thematic model, it also includes: S800: Extracts historical and current data based on a comprehensive water affairs thematic model.

[0135] Historical data refers to data generated within a pre-defined historical timeframe prior to the current monitoring time in the integrated water management thematic model. Current data refers to data generated at the current monitoring time in the integrated water management thematic model.

[0136] Based on the unique business identifier, business theme, data fields, and timestamp in the integrated water affairs theme model, data within a pre-defined historical time range prior to the current detection time is extracted as historical data, and the latest data at the current detection time is extracted as current data.

[0137] S801: Determine the dynamic tolerance trend range using historical data.

[0138] The dynamic tolerance trend range refers to the allowable offset range calculated based on the normal variation patterns of historical data within a preset historical time range, used to determine whether the current offset is within a reasonable fluctuation range. The dynamic tolerance trend range includes a lower limit and an upper limit for dynamic offset.

[0139] Extract the most recent U historical data points arranged chronologically from the historical data, and calculate the trend baseline value T based on these U historical data points. Here, U is the preset number of trend analysis windows, and the trend baseline value T refers to the current theoretical reference value predicted based on the changing trends of historical data. Then, calculate the historical offset of each of the U historical data points relative to its respective trend reference value, forming a historical offset sequence. Determine the offset fluctuation value D based on the historical offset sequence, and combine it with a preset trend tolerance coefficient q to calculate the dynamic offset lower limit value as −qD and the dynamic offset upper limit value as +qD, thus forming a dynamic tolerance trend range. The offset fluctuation value D can be the mean absolute deviation, residual standard deviation, or preset quantile offset value of the historical offset sequence.

[0140] S802: Calculate and determine the current offset by combining historical data and current data.

[0141] The current offset refers to the degree of deviation of the current data from the historical baseline.

[0142] The current offset Δ = xt is obtained by calculating the difference between the current data x and the historical baseline value t.

[0143] S803: If the current offset is within the dynamic tolerance trend range, jump to execute the data acquisition call request.

[0144] If the current offset Δ is greater than or equal to the lower limit of the dynamic offset and less than or equal to the upper limit of the dynamic offset, it indicates that the current data does not have abnormal fluctuations relative to the historical normal change pattern, maintains the callable state of the integrated water affairs theme model, and executes the data call process when a data call request is received.

[0145] S804: If the current offset is not within the dynamic tolerance trend range, obtain the association path between the fields of each table during the horizontal splicing process of the standard data detail table.

[0146] The association path refers to the pre-recorded field source relationships, field mapping relationships, inter-table connection relationships, or field transformation relationships when performing horizontal splicing of standard data detail tables.

[0147] If the current offset Δ is less than the lower limit of the dynamic offset or greater than the upper limit of the dynamic offset, it indicates that the current data has abnormal fluctuations relative to the historical normal change pattern. The association path corresponding to the abnormal field is obtained from the field lineage record. The field lineage record refers to the record formed when S603 performs horizontal splicing, which records the source business system, source table name, source field name, business unique identifier, splicing primary key, field mapping rules, and field conversion rules corresponding to the target field.

[0148] S805: Based on the associated path, determine all corresponding associated nodes, and combine the associated nodes and associated path to determine the splicing node.

[0149] A related node refers to a data object node involved in a related path, which at least includes a standard data detail table, fields, a business unique identifier, and a source business system or data processing step. A concatenation node refers to a key node used to connect different standard data detail tables during the horizontal concatenation process.

[0150] Based on the association path, extract the standard data detail tables, fields, business unique identifiers and source business systems involved in the association path as association nodes; then determine the business unique identifiers, connection fields or unified primary keys of the middle platform used to connect different standard data detail tables from the association nodes as splicing nodes.

[0151] S806: Combine splicing nodes and associated paths to determine the data association graph.

[0152] A data association graph is a graph structure used to represent the relationships between different standard data detail tables, fields, business unique identifiers, source business systems, and data processing steps.

[0153] By using associated nodes as graph nodes, associated paths as graph edges, and marking splicing nodes as key connection nodes, a data association graph is constructed.

[0154] S807: Based on the data association graph, reverse trace the corresponding current data to identify the abnormal target and output the result.

[0155] An abnormal target refers to the source business system, standard data detail table, field, business object, acquisition device, or data processing step that causes the current data to deviate from the dynamic tolerance trend range.

[0156] Based on the data association graph, the current abnormal data is used as the starting point for tracing. The reverse tracing is carried out along the association path to determine the splicing node, association field, standard data detail table and source business system corresponding to the current data in sequence. Based on the reverse tracing results, the abnormal target that caused the current data abnormality is determined, and the abnormal target and its corresponding association path, abnormal field, data source and abnormal cause are output.

[0157] The integrated water management model is encapsulated into a standard application interface and output, specifically including: S900: Receives data access requests and extracts the security access token from the data access requests.

[0158] A data retrieval request is an access request initiated by a requester to access data content within an integrated water management thematic model. A security access token is verification information carried in a data retrieval request, used to identify the requester's identity or access eligibility.

[0159] By receiving data access requests from the requesting party, a security access token can be extracted from the request header, request parameters, or request body. The requesting party can be an external business system, an internal business system, a data analysis terminal, or a user terminal with data access needs.

[0160] S901: Match and verify the security access token with the preset permission list to determine the interface access permissions.

[0161] A pre-configured permission list refers to a pre-configured data table that records the correspondence between security access tokens, requester identity, accessible interfaces, permission levels, validity periods, access counts, and access scopes. Interface access permissions refer to whether the requester has the authority to invoke the target standard application interface.

[0162] The security access token is entered into the permission list for matching and verification. It is determined whether the security access token exists, whether it is within the validity period, whether the number of accesses has not exceeded the limit, and whether the security access token has the permission to call the target standard application interface. The interface access permissions are determined based on the matching and verification results.

[0163] S902: When the interface access permission is "permitted", obtain the access level corresponding to the data call request based on the interface access permission.

[0164] Access level refers to the specific level at which a requester, given interface access permissions, can access the data content in the integrated water affairs thematic model.

[0165] When the interface access permission is granted, the access level corresponding to the data call request is determined based on the requester's identity, accessible interface, permission level, access scope, and validity period corresponding to the security access token in the preset permission list.

[0166] S903: Retrieve viewable content of the corresponding permission level in the integrated water affairs theme model as access content based on the access level.

[0167] Access levels refer to the levels of access assigned to fields, records, or business themes in a comprehensive water management thematic model based on data sensitivity, business roles, application system types, or data usage scope. Accessed content refers to the data content that is allowed to be viewed, accessed, or output under the corresponding access level.

[0168] Based on the access level, data fields, records, tables, or business themes matching the access level are selected from the integrated water affairs thematic model as the access content. The access content can differ for different access levels; lower access levels correspond to summary data or anonymized data, while higher access levels correspond to detailed data or complete field data.

[0169] S904: Encapsulate and output the accessed content through the standard application interface.

[0170] A standard application interface refers to a unified interface used to output accessed content to the requesting party.

[0171] The filtered access content is encapsulated according to a preset interface message format and output to the requester through a standard application interface. The preset interface message format includes JSON, XML, Protobuf, or other structured message formats.

[0172] S905: When the interface access permission is no, obtain the application reason based on the data call request, approve the application based on the application reason, and obtain the approval result.

[0173] The reason for the application refers to the information submitted by the requesting party when it does not have access to the interface, explaining the purpose, scope, duration, or business requirements of the data call. The approval result refers to the result of approving or disapproving the application reason.

[0174] When the interface access permission is no, the application reason is extracted from the data call request, and the application reason, the identity of the requester, the target interface, the scope of the request, and the duration of the request are submitted to the preset approval process for approval to obtain the approval result.

[0175] S906: If the approval result is passed, a temporary access token, temporary permissions, and temporary access level are generated based on the approval result. After filtering the corresponding access content based on the temporary access level, the content is output through the standard application interface.

[0176] A temporary access token is a verification message generated when the approval result is passed, granting temporary access. Temporary permissions refer to interface access permissions temporarily granted to the requesting party based on the approval result. Temporary access level refers to the data access level corresponding to the temporary permissions.

[0177] If the approval result is passed, a temporary access token is generated based on the approval result, and temporary permissions and temporary access levels are configured. The filtered access content is then encapsulated and output through a standard application interface.

[0178] S907: If the approval result is not approved, no access content will be output, and an access denial message will be generated.

[0179] If the approval result is not approved, the access content in the integrated water affairs theme model will not be output to the requester. The reason for rejection will be determined through the approval result, and then the reason for rejection, the target interface, the identity of the requester, and the request time will be combined as the access rejection information.

[0180] Based on the same inventive concept, embodiments of the present invention provide an AI-based water data quality processing system, comprising: The data acquisition module is used to collect data retrieval requests. The memory stores a program for implementing an AI-based control method for water data quality processing as described in any one of the above descriptions; The processor loads and executes programs stored in memory.

[0181] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0182] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A water data quality processing method based on AI, characterized in that, include: Establish a data monitoring channel, a data standard verification rule base, and a data tiered area consisting of multiple storage levels. Obtain initial water affairs data according to the data monitoring channel and write it into the data buffer pool as a data stream to be inspected. Extract target business data items from the data stream to be inspected; Determine the numerical range thresholds and format conditions based on the data standard verification rule base; When any target business data item does not meet its corresponding format conditions or numerical range threshold, the corresponding data stream to be inspected is intercepted, and all data streams to be inspected that pass the comparison are marked as qualified water affairs data. Qualified water affairs data are mapped and allocated to data hierarchical areas, and corresponding standard data detail tables are constructed. Extract the business unique identifier from the corresponding standard data detail table, and concatenate the standard data detail tables with the same business unique identifier across different standard data detail tables to generate a comprehensive water affairs theme model. Data collection request; Based on the data retrieval request, the integrated water affairs theme model is encapsulated into a standard application interface and output.

2. The AI-based water data quality processing method according to claim 1, characterized in that, Initial water data obtained from the data monitoring channel includes: Obtain underlying business data; Establish a connection between the data monitoring channel and the underlying business data, and obtain the change log file in the underlying business data; Extract the operation content recorded in the change log file, and extract the corresponding timestamp and business source identifier based on the operation content; The timestamp, business source identifier, and operation content are encapsulated as initial water affairs data.

3. The AI-based water data quality processing method according to claim 2, characterized in that, Writing to the data buffer pool includes: The data buffer pool is configured as a multi-node distributed cluster; Parse the initial water data, extract the timestamp and business source identifier, and concatenate the timestamp and business source identifier as the routing key; A preset routing distribution algorithm is executed on the routing key to calculate and determine the corresponding target node partition index; Based on the target node partition index, the initial water data is distributed and written to the corresponding multi-node distributed cluster in the data buffer pool for queuing and buffering.

4. The AI-based water data quality processing method according to claim 1, characterized in that, Intercepting the corresponding data streams to be inspected, and marking all data streams that pass the comparison as qualified water data includes: Extract blank rules and format matching rules for the target business data item from the data standard verification rule library as format conditions, and extract the historical normal operation data of the target business data item; The threshold value range is determined based on historical normal operation data; Extract the data content format and data values ​​based on the target business data items; When the data content format does not meet the format conditions, or the data value exceeds the value range threshold, an interception command is generated for the data stream to be inspected, and the data stream is intercepted. When the data content format meets the format conditions and the data value is within the threshold of the value range, the data stream to be inspected is marked as qualified water affairs data.

5. The AI-based water data quality processing method according to claim 4, characterized in that, After interception, the following is included: Define the corresponding target business data item as abnormal data and write it to the preset abnormal data station in real time; Analyze the abnormal data in the abnormal data station to extract the error causes and data sources; The cause of the error is fed back from the data source to obtain correction information, and the correction information is then used as the target business data item again. The number of corrections is obtained based on the correction information; Determine the allowed number of times based on the blank rule; If the number of corrections exceeds the target allowed number, the corresponding abnormal data is obtained, treated as red line data, and output.

6. The AI-based water data quality processing method according to claim 1, characterized in that, The qualified water data is mapped and allocated to construct a standard data detail table and generate a comprehensive water theme model, specifically including: In the pre-defined storage database, the technology buffer layer, near-source model layer, integration model layer and application mart layer are divided and isolated from each other according to the technology buffer, near-source model, integration model and application mart stages of data flow. Qualified water data is input into the technical buffer layer, and then distributed and loaded into the near-source model layer by the technical buffer layer, and converted into a standard data detail table; Based on the unique business identifier, the standard data detail tables of different business systems are spliced ​​across tables in the integration model layer to generate a comprehensive water affairs theme model, which is then solidified and loaded into the application mart layer.

7. The AI-based water data quality processing method according to claim 6, characterized in that, Cross-table concatenation of standard data detail tables belonging to different business systems includes: Extract the original subsystem numbers from the standard data detail tables belonging to different business systems; The unified primary key of the middle platform is calculated and generated based on the current system time and server node number, and a global code lookup table is constructed. The original numbers of each subsystem are extracted and entered into the global code lookup table for retrieval and matching. The original numbers of each subsystem are then uniformly replaced with the corresponding unified primary key of the middle platform. Update the corresponding standard data detail table with the same unified primary key of the middle platform as the corresponding business unique identifier; For standard data detail tables belonging to different business systems, but having the same unique business identifier as the basis for cross-table concatenation, perform horizontal concatenation operations.

8. The AI-based water data quality processing method according to claim 6, characterized in that, After generating the integrated water affairs thematic model, it also includes: Historical and current data are extracted based on a comprehensive water affairs thematic model; Determine the dynamic tolerance trend range using historical data; The current offset is determined by combining historical and current data. If the current offset is within the dynamic tolerance trend range, jump to execute the data collection request; If the current offset is not within the dynamic tolerance trend range, obtain the association path between the fields of each table during the horizontal splicing process of the standard data detail table; Based on the associated path, determine all corresponding associated nodes, and combine the associated nodes and associated path to determine the splicing node; The data association graph is determined by combining splicing nodes and associated paths; Based on the data association graph, the corresponding current data is traced back to identify abnormal targets, and the results are output.

9. The AI-based water data quality processing method according to claim 1, characterized in that, The integrated water management model is encapsulated into a standard application interface and output, specifically including: Receive data retrieval requests and extract the security access token from the data retrieval requests; The security access token is matched and verified against the preset permission list to determine the access permissions of the interface. When the interface access permission is granted, the access level corresponding to the data call request is obtained based on the interface access permission. Based on the access level, retrieve the viewable content corresponding to the access level in the integrated water affairs theme model as the access content; The accessed content is encapsulated and output through standard application interfaces; When the interface access permission is no, the application reason is obtained from the data call request, the application reason is approved, and the approval result is obtained. If the approval result is passed, a temporary access token, temporary permissions, and temporary access level are generated based on the approval result. The corresponding access content is then filtered based on the temporary access level and output through the standard application interface. If the approval result is negative, no access content will be output, and an access denial message will be generated.

10. An AI-based water data quality processing system, characterized in that, include: The data acquisition module is used to collect data retrieval requests. The memory stores a program for implementing an AI-based control method for water data quality processing as described in any one of claims 1 to 9; The processor loads and executes programs stored in memory.

Images