A wastewater purification data processing method

Abstract

This invention discloses a wastewater purification data processing method, belonging to the field of data processing technology, comprising the following steps: S1, obtaining the existing index of the wastewater purification data package, and determining the transmission set between the latest data report and each leaf node of the existing index when generating the latest data report; S2, generating the optimal leaf node based on the transmission set between the latest data report and each leaf node of the existing index; S3, inserting the latest data report into the optimal leaf node to complete the update of the wastewater purification data package. This invention considers the node fill factor when selecting the optimal leaf node, prioritizing nodes with higher fill factors to reduce storage space waste. Simultaneously, by balancing the splitting frequency and overflow risk, it can improve the utilization rate of storage resources while ensuring the balance of the data structure.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and specifically relates to a wastewater purification data processing method. Background Technology

[0002] In wastewater treatment, data acquisition, processing, and analysis are crucial for ensuring treatment effectiveness, optimizing process parameters, and achieving intelligent management. However, traditional wastewater treatment data processing methods often face the following challenges: First, the data volume is massive and complex. Wastewater treatment systems involve multiple monitoring points (such as inlets, outlets, and reaction tanks), each generating a large amount of data in real time (such as pH values, COD concentrations, ammonia nitrogen content, and flow rates). This data is not only vast in scale but also characterized by high dimensionality, high frequency, and multi-source heterogeneity, placing enormous pressure on data storage, management, and analysis. Second, data indexing and retrieval efficiency is low. In traditional data processing methods, wastewater treatment data is typically stored in a linear or simple tree structure, lacking an efficient indexing mechanism. When querying or updating data for a specific time period or a specific monitoring point, the system often needs to traverse a large amount of data, resulting in low retrieval efficiency and failing to meet real-time requirements. Furthermore, as the wastewater treatment system operates, new data reports are continuously generated (such as daily reports and real-time monitoring reports). Traditional methods often lack intelligent node selection strategies when inserting new data into existing data structures, which may lead to uneven data distribution, storage redundancy, or decreased query performance. Summary of the Invention

[0003] To address the above problems, this invention proposes a wastewater purification data processing method.

[0004] The technical solution of the present invention is: a wastewater purification data processing method comprising the following steps:

[0005] S1. Obtain the existing index of the wastewater purification data package. When generating the latest data report, determine the transmission set between the latest data report and each leaf node of the existing index.

[0006] S2. Generate the optimal leaf node based on the transfer set between the latest data report and each leaf node of the existing index;

[0007] S3. Insert the latest data report into the optimal leaf node to complete the update of the wastewater purification data package.

[0008] Furthermore, S1 includes the following sub-steps:

[0009] S11. Obtain the existing index of the wastewater purification data package; the existing index is a B+ tree index;

[0010] S12. When inserting the latest data report into the wastewater purification data package, obtain the primary key ID of the latest data report;

[0011] S13. Based on the primary key ID of the latest data report, determine the transfer set between the latest data report and each leaf node of the B+ tree index.

[0012] The beneficial effects of the above-mentioned further solutions are as follows: In this invention, the B+ tree is a multi-way balanced search tree with efficient retrieval and range query capabilities. In wastewater purification data processing, the B+ tree index can quickly locate data with a specific primary key ID. By uniquely identifying each data report through the primary key ID (such as a timestamp, sensor ID, etc.), the data that needs to be processed can be quickly located, improving the efficiency of data processing. By analyzing the transmission set between the latest data report and the B+ tree index nodes, the distribution characteristics of the data can be dynamically adapted, and the optimal insertion node can be selected, thereby maintaining the balance and efficiency of the data structure.

[0013] Furthermore, S13 includes the following sub-steps:

[0014] S131. Generate random numbers;

[0015] S132. Based on the primary key ID and random number of the latest data report, determine the transmission set between the latest data report and the root node of the B+ tree index;

[0016] S133. Based on the transfer set between the latest data report and the root node of the B+ tree index, determine the transfer set between the latest data report and each leaf node of the B+ tree index.

[0017] The beneficial effects of the above-mentioned further solutions are as follows: In this invention, randomness is introduced when determining the transmission set by generating random numbers, which helps to avoid excessive concentration of data on certain nodes and further optimizes the data distribution. By first determining the transmission set between the latest data report and the root node, and then deriving the transmission set between the report and the leaf nodes, a step-by-step analysis from top to bottom is achieved, ensuring the accuracy and efficiency of the transmission set.

[0018] Furthermore, in S132, the transfer set X between the latest data report and the root node of the B+ tree index root The expression is: In the formula, p represents a random number, M represents the number of key-value pairs stored in the root node, and L represents the number of characters contained in the primary key ID of the latest data report. This indicates the rounding up operation.

[0019] Furthermore, in S133, the expression for the transfer set between the latest data report and the i-th leaf node of the B+ tree index is:

[0020] In the formula, N1 represents the number of key-value pairs stored in the first leaf node, N2 represents the number of key-value pairs stored in the second leaf node, and N... i This represents the number of key-value pairs stored in the i-th leaf node. This indicates a rounding up operation, p represents a random number, and L represents the number of characters in the primary key ID of the latest data report.

[0021] The beneficial effects of the above-mentioned further solutions are as follows: In this invention, by precisely quantizing the transmission set, the optimal position for data insertion can be scientifically determined based on parameters such as primary key ID, random number, and the number of node key values. Considering the dynamic changes in data distribution (such as changes in the number of node key values), the transmission set can be dynamically adjusted according to the actual situation, maintaining the balance of the data structure.

[0022] Furthermore, S2 includes the following sub-steps:

[0023] S21. Based on the transmission set between each leaf node of the latest data report and the existing index, determine the alternation scheduling index between adjacent leaf nodes;

[0024] S22. Extract the two adjacent leaf nodes corresponding to the maximum alternating scheduling index;

[0025] S23. Among two adjacent leaf nodes, select the leaf node with the largest fill factor as the best leaf node.

[0026] The beneficial effects of the above-mentioned further scheme are as follows: In this invention, by calculating the alternation scheduling index between adjacent leaf nodes, which considers the maximum, minimum, and median of the transmission set, the alternation scheduling index can more comprehensively reflect the correlation between data and nodes. The design of the alternation scheduling index helps to avoid excessive concentration of data on certain nodes, thereby reducing data skew problems. By extracting the two adjacent leaf nodes corresponding to the maximum alternation scheduling index, the method can accurately locate the candidate node most likely to be the best insertion position. Selecting the adjacent leaf node with the largest alternation scheduling index as a candidate, and selecting the leaf node with the largest fill factor as the best leaf node, can optimize storage utilization. The fill factor represents the actual proportion of key values ​​or data occupied in a node; selecting a node with a larger fill factor can reduce the waste of storage space. Furthermore, a high fill factor (such as 80%) can reduce the frequency of splits.

[0027] Furthermore, in S21, the alternation scheduling exponent w between the i-th leaf node and the (i+1)-th leaf node... i,i+1 The calculation formula is:

[0028] In the formula, This represents the maximum number of sets of data transfers between the latest data report and the i-th leaf node of the B+ tree index. This represents the minimum number of transfer sets between the latest data report and the i-th leaf node of the B+ tree index. This represents the median of the transfer set between the latest data report and the i-th leaf node of the B+ tree index. This represents the maximum number of transfer sets between the latest data report and the (i+1)th leaf node of the B+ tree index. This represents the minimum number of transfer sets between the latest data report and the (i+1)th leaf node of the B+ tree index. This represents the median of the transfer set between the latest data report and the (i+1)th leaf node of the B+ tree index. This indicates a round-up operation. This indicates the operation of retrieving the maximum value. This indicates the operation of taking the minimum value.

[0029] The beneficial effects of this invention are:

[0030] (1) The present invention uses B+ tree as index structure, which significantly improves the data retrieval efficiency. The multi-way balancing characteristic of B+ tree ensures the uniform distribution of data among nodes, which is suitable for the processing needs of a large amount of real-time data in sewage purification system.

[0031] (2) By calculating the alternation scheduling index between adjacent leaf nodes, the present invention can dynamically sense changes in data distribution and adjust the data insertion strategy accordingly, thus being able to cope with dynamic changes in the amount of data in the sewage purification system.

[0032] (3) When selecting the best leaf node, the present invention takes into account the node's fill factor and prioritizes nodes with higher fill factors to reduce the waste of storage space. At the same time, by balancing the split frequency and the risk of overflow, it can improve the utilization rate of storage resources while ensuring the balance of the data structure. Attached Figure Description

[0033] Figure 1 This is a flowchart of a wastewater purification data processing method. Detailed Implementation

[0034] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0035] like Figure 1 As shown, the present invention provides a wastewater purification data processing method, comprising the following steps:

[0036] S1. Obtain the existing index of the wastewater purification data package. When generating the latest data report, determine the transmission set between the latest data report and each leaf node of the existing index.

[0037] S2. Generate the optimal leaf node based on the transfer set between the latest data report and each leaf node of the existing index;

[0038] S3. Insert the latest data report into the optimal leaf node to complete the update of the wastewater purification data package.

[0039] The structured storage of wastewater treatment data packages uses a B+ tree index structure. It can include basic attribute fields and dynamic monitoring data. Basic attribute fields include wastewater treatment plant ID (primary key), name, type (above-ground / underground), geographical location (longitude, latitude), designed treatment capacity, process type (e.g., A² / O, MBR), and discharge standards. Dynamic monitoring data includes water quality indicators such as chemical oxygen demand (COD), ammonia nitrogen, total phosphorus, and pH, as well as flow data such as influent flow rate, effluent flow rate, and sludge return flow rate.

[0040] The latest data reports are linked to wastewater treatment data packages via primary key IDs and can be in CSV format or other formats. They can include records of abnormal events and trend analysis results.

[0041] In this embodiment of the invention, S1 includes the following sub-steps:

[0042] S11. Obtain the existing index of the wastewater purification data package; the existing index is a B+ tree index;

[0043] S12. When inserting the latest data report into the wastewater purification data package, obtain the primary key ID of the latest data report;

[0044] S13. Based on the primary key ID of the latest data report, determine the transfer set between the latest data report and each leaf node of the B+ tree index.

[0045] In this invention, the B+ tree is a multi-way balanced search tree with efficient retrieval and range query capabilities. In wastewater treatment data processing, the B+ tree index can quickly locate data with a specific primary key ID. By uniquely identifying each data report using a primary key ID (such as a timestamp, sensor ID, etc.), the data requiring processing can be quickly located, improving data processing efficiency. By analyzing the transmission set between the latest data report and the B+ tree index nodes, the data distribution characteristics can be dynamically adapted, selecting the optimal insertion node, thereby maintaining the balance and efficiency of the data structure.

[0046] In this embodiment of the invention, S13 includes the following sub-steps:

[0047] S131. Generate random numbers;

[0048] S132. Based on the primary key ID and random number of the latest data report, determine the transmission set between the latest data report and the root node of the B+ tree index;

[0049] S133. Based on the transfer set between the latest data report and the root node of the B+ tree index, determine the transfer set between the latest data report and each leaf node of the B+ tree index.

[0050] In this invention, randomness is introduced when determining the transmission set by generating random numbers, which helps to avoid excessive concentration of data on certain nodes and further optimizes the data distribution. By first determining the transmission set between the latest data report and the root node, and then deriving the transmission set between the report and the leaf nodes, a step-by-step analysis from top to bottom is achieved, ensuring the accuracy and efficiency of the transmission set.

[0051] In this embodiment of the invention, in S132, the transmission set X between the latest data report and the root node of the B+ tree index... root The expression is:

[0052] In the formula, p represents a random number, M represents the number of key-value pairs stored in the root node, and L represents the number of characters contained in the primary key ID of the latest data report. This indicates the rounding up operation.

[0053] In this embodiment of the invention, in S133, the expression for the transmission set between the latest data report and the i-th leaf node of the B+ tree index is:

[0054] In the formula, N1 represents the number of key-value pairs stored in the first leaf node, N2 represents the number of key-value pairs stored in the second leaf node, and N... i This represents the number of key-value pairs stored in the i-th leaf node. This indicates a rounding up operation, p represents a random number, and L represents the number of characters in the primary key ID of the latest data report.

[0055] In this invention, by precisely quantizing the transmission set, the optimal position for data insertion can be scientifically determined based on parameters such as primary key ID, random number, and the number of node key values. It also considers dynamic changes in data distribution (such as changes in the number of node key values) and can dynamically adjust the transmission set according to actual conditions to maintain the balance of the data structure.

[0056] In this embodiment of the invention, S2 includes the following sub-steps:

[0057] S21. Based on the transmission set between each leaf node of the latest data report and the existing index, determine the alternation scheduling index between adjacent leaf nodes;

[0058] S22. Extract the two adjacent leaf nodes corresponding to the maximum alternating scheduling index;

[0059] S23. Among two adjacent leaf nodes, select the leaf node with the largest fill factor as the best leaf node.

[0060] In this invention, by calculating the alternation scheduling index between adjacent leaf nodes, which considers the maximum, minimum, and median of the transmission set, the correlation between data and nodes can be more comprehensively reflected. The design of the alternation scheduling index helps avoid excessive data concentration on certain nodes, thereby reducing data skew problems. By extracting the two adjacent leaf nodes corresponding to the maximum alternation scheduling index, the method can accurately locate the candidate node most likely to be the optimal insertion position. Selecting the adjacent leaf node with the largest alternation scheduling index as a candidate, and then selecting the leaf node with the largest fill factor as the optimal leaf node, optimizes storage utilization. The fill factor represents the actual proportion of key-value pairs or data occupied in a node; selecting a node with a larger fill factor can reduce storage space waste. Furthermore, a high fill factor (e.g., 80%) can reduce the frequency of splits.

[0061] In this embodiment of the invention, in S21, the alternation scheduling index w between the i-th leaf node and the (i+1)-th leaf node is... i,i+1 The calculation formula is:

[0062] In the formula, This represents the maximum number of sets of data transfers between the latest data report and the i-th leaf node of the B+ tree index. This represents the minimum number of transfer sets between the latest data report and the i-th leaf node of the B+ tree index. This represents the median of the transfer set between the latest data report and the i-th leaf node of the B+ tree index. This represents the maximum number of transfer sets between the latest data report and the (i+1)th leaf node of the B+ tree index. This represents the minimum number of transfer sets between the latest data report and the (i+1)th leaf node of the B+ tree index. This represents the median of the transfer set between the latest data report and the (i+1)th leaf node of the B+ tree index. This indicates a round-up operation. This indicates the operation of retrieving the maximum value. This indicates the operation of taking the minimum value.

[0063] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A wastewater purification data processing method, characterized in that, Includes the following steps: S1. Obtain the existing index of the wastewater purification data package. When generating the latest data report, determine the transmission set between the latest data report and each leaf node of the existing index. S2. Generate the optimal leaf node based on the transfer set between the latest data report and each leaf node of the existing index; S3. Insert the latest data report into the best leaf node to complete the update of the wastewater purification data package; S1 includes the following sub-steps: S11. Obtain the existing index of the wastewater purification data package; the existing index is a B+ tree index; S12. When inserting the latest data report into the wastewater purification data package, obtain the primary key ID of the latest data report; S13. Based on the primary key ID of the latest data report, determine the transfer set between the latest data report and each leaf node of the B+ tree index; S13 includes the following sub-steps: S131. Generate random numbers; S132. Based on the primary key ID and random number of the latest data report, determine the transmission set between the latest data report and the root node of the B+ tree index; S133. Based on the transfer set between the latest data report and the root node of the B+ tree index, determine the transfer set between the latest data report and each leaf node of the B+ tree index. S2 includes the following sub-steps: S21. Based on the transmission set between each leaf node of the latest data report and the existing index, determine the alternation scheduling index between adjacent leaf nodes; S22. Extract the two adjacent leaf nodes corresponding to the maximum alternating scheduling index; S23. Among two adjacent leaf nodes, select the leaf node with the largest fill factor as the best leaf node.

2. The wastewater purification data processing method according to claim 1, characterized in that, In step S132, the transmission set X between the latest data report and the root node of the B+ tree index... root The expression is: In the formula, p represents a random number, M represents the number of key-value pairs stored in the root node, and L represents the number of characters contained in the primary key ID of the latest data report. This indicates the rounding up operation.

3. The wastewater purification data processing method according to claim 1, characterized in that, In step S133, the expression for the transmission set between the latest data report and the i-th leaf node of the B+ tree index is: In the formula, N1 represents the number of key-value pairs stored in the first leaf node, N2 represents the number of key-value pairs stored in the second leaf node, and N... i This represents the number of key-value pairs stored in the i-th leaf node. This indicates a rounding up operation, p represents a random number, and L represents the number of characters in the primary key ID of the latest data report.

4. The wastewater purification data processing method according to claim 1, characterized in that, In S21, the alternation scheduling index w between the i-th leaf node and the (i+1)-th leaf node is... i,i+1 The calculation formula is: In the formula, This represents the maximum number of sets of data transfers between the latest data report and the i-th leaf node of the B+ tree index. This represents the minimum number of transfer sets between the latest data report and the i-th leaf node of the B+ tree index. This represents the median of the transfer set between the latest data report and the i-th leaf node of the B+ tree index. This represents the maximum number of transfer sets between the latest data report and the (i+1)th leaf node of the B+ tree index. This represents the minimum number of transfer sets between the latest data report and the (i+1)th leaf node of the B+ tree index. This represents the median of the transfer set between the latest data report and the (i+1)th leaf node of the B+ tree index. This indicates a round-up operation. This indicates the operation of retrieving the maximum value. This indicates the operation of taking the minimum value.

Images