Urban water resource detection method and device based on sparse crowd sensing and medium
Through the urban water resource detection method based on sparse group intelligence perception, portable water quality detection equipment and BiLSTM are used to reconstruct incomplete water quality data, which solves the problems of high detection cost and low coverage in large-scale irregular water quality detection areas, and achieves high-precision and low-cost water quality detection effects.
Patent Information
- Application Number
- CN202411922455.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-25
- Publication Date
- 2025-05-30
AI Technical Summary
The prior art has problems such as high detection cost, low coverage and inaccurate detection results in large-scale irregular water quality detection areas.
The urban water resource detection method based on sparse group intelligence perception is adopted, and incomplete water quality data is reconstructed through portable water quality detection equipment and bidirectional long and short-term memory network (BiLSTM), achieving high coverage and low-cost water quality detection.
It realizes high coverage and low cost water quality detection, improves data reconstruction accuracy, can handle irregular sensing areas, and maintains noise immunity and robustness under high data loss rates and noise conditions.
Smart Images
Figure CN120067662A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of urban water resource detection, and particularly to a method, device and medium for urban water resource detection based on sparse crowdsensing. Background Art
[0002] The rapid development of cities has led to an increasingly serious problem of water resource pollution. Traditional water quality detection mainly relies on the Internet of Things and sensor networks, and has made remarkable progress in data accuracy and collection efficiency. However, for large-scale irregular water quality detection areas, how to accurately detect water quality information in a low-cost and high-coverage manner remains an urgent problem to be solved.
[0003] Mobile Crowdsensing (MCS) uses personal devices as sensing nodes and can achieve wide-range environmental data collection at low cost. It has been widely used in fields such as noise level sensing and traffic detection. In the field of water quality detection, there is less research based on MCS, mainly because there is a lack of portable instruments for detecting water quality parameters, and for vast and remote water areas, the detection difficulty and cost are relatively high.
[0004] In recent years, the popular application of portable water quality detection instruments and the proposal of the Sparse Mobile Crowdsensing (SMCS) framework have provided new research ideas for water quality detection based on MCS. The SMCS method utilizes the spatio-temporal correlation of data and can reconstruct data through a small amount of sensing data, such as using compressive sensing, matrix completion, and interpolation methods based on neighboring nodes, etc., so as to effectively reduce the number of sensing tasks and the detection cost, and has been applied in large-scale regional environmental detections such as air quality and noise level. However, due to the complex shape of the water quality detection area and the high data loss rate (the proportion of units without collection tasks in the total task units), the traditional data reconstruction algorithms in SMCS are difficult to meet the requirements of environmental protection departments for detection accuracy. Therefore, there is currently no relevant research on using the SMCS method for urban water resource detection. Summary of the Invention
[0005] The purpose of the present invention is to propose a method, device and medium for urban water resource detection based on sparse crowdsensing, so as to solve the technical problems of high detection cost, low coverage rate and inaccurate detection results existing in the prior art when detecting water quality in large-scale irregular water quality detection areas.
[0006] A method for urban water resource detection based on sparse crowdsensing provided by the present invention includes the following steps:
[0007] S1: Construct a model for urban water resource detection based on sparse crowdsensing;
[0008] S2: Solve the urban water resource detection model based on sparse crowd sensing using the trained neural network to obtain all water resource detection data in the detection area.
[0009] A storage medium stores instructions and data for implementing a method for detecting urban water resources based on sparse crowd sensing.
[0010] An urban water resource detection device based on sparse crowd sensing includes: a processor and the storage medium; the processor loads and executes the instructions and data in the storage medium for implementing a method for detecting urban water resources based on sparse crowd sensing.
[0011] The beneficial effects provided by the present invention are:
[0012] (1) High-coverage and low-cost water quality detection: With the help of portable water quality detection equipment, through the sparse crowd sensing method, comprehensive detection of a wide range of waters is achieved using a small number of collection points, overcoming the problems of low coverage and high cost of traditional methods.
[0013] (2) Improvement in data reconstruction accuracy: The bidirectional long short-term memory network (BiLSTM) is used to reconstruct the incomplete water quality data collected. Compared with traditional reconstruction algorithms and other deep learning models, BiLSTM shows higher accuracy and stability, especially in the case of a high data loss rate.
[0014] (3) Ability to handle irregular sensing areas: Aiming at the problems of complex shapes of water area detection areas and limited collection conditions, the proposed model can handle the irregularity of the sensing area, and the reconstruction results meet the requirements of environmental protection departments for water quality detection accuracy.
[0015] (4) Noise resistance and robustness: In the face of a high data loss rate and noise in the sensed data, the model can better filter out the noise and compensate for the missing data, ensuring the reliability and stability of the detection results. Description of the Drawings
[0016] Figure 1 is a schematic diagram of the simple process of the method of the present invention;
[0017] Figure 2 is a schematic diagram of the data matrix;
[0018] Figure 3 is a schematic diagram of the environmental matrix;
[0019] Figure 4 is a schematic diagram of the sensing matrix;
[0020] Figure 5 is a schematic diagram of the reconstruction matrix;
[0021] Figure 6 It is a schematic diagram of the error rate of the BiLSTM model at different epochs when the data loss rate is fixed at 60%;
[0022] Figure 7 It is a graph showing the relationship between the data loss rate and the error rate with the pH value as the detection index when comparing the method of the present invention with the traditional data reconstruction algorithm in the first group of embodiments of the present invention;
[0023] Figure 8 It is a graph showing the relationship between the data loss rate and the error rate with the oxygen content as the detection index when comparing the method of the present invention with the traditional data reconstruction algorithm in the first group of embodiments of the present invention;
[0024] Figure 9 It is a graph showing the relationship between the data loss rate and the error rate with the pH value as the detection index when comparing the method of the present invention with other deep learning models in the second group of embodiments of the present invention;
[0025] Figure 10 It is a graph showing the relationship between the data loss rate and the error rate with the oxygen content as the detection index when comparing the method of the present invention with other deep learning models in the second group of embodiments of the present invention;
[0026] Figure 11 It is a schematic diagram of the operation of the hardware device of the present invention. Detailed implementation manners
[0027] To make the objectives, technical solutions and advantages of the present invention clearer, the embodiments of the present invention will be further described below in conjunction with the accompanying drawings.
[0028] Before formally elaborating on the present invention, a general description of the solution of the present invention is first given for easy understanding.
[0029] Please refer to Figure 1 , the present invention provides a method for detecting urban water resources based on sparse crowd sensing, including the following steps:
[0030] S1: Construct a model for detecting urban water resources based on sparse crowd sensing;
[0031] It should be noted that step S1 is specifically as follows:
[0032] S11. Study the investigation reports on urban water resource pollution in multiple places to determine the pollution sources;
[0033] As an embodiment, after studying the investigation reports on urban water resource pollution in multiple places in the present invention, it is found that the main reason for urban water resource pollution lies in the pollution along the water area. After these pollutants enter the water body, after a period of diffusion, they gradually affect the water quality of the entire water area.
[0034] S12. Take the pollution source area as the sensing area, and divide the sensing area into multiple sensing units of the same size;
[0035] As an embodiment, since water pollution sources basically appear on the shore, the present invention expands the sensing area along the water body shore and divides the sensing area into sensing units of the same size. Select some sensing units to distribute water resource detection tasks, and use data reconstruction algorithms to infer the overall data after receiving the water resource detection data. During the process of urban water resource detection using the sparse crowd-sourced sensing method, the total area to be detected needs to be divided into many sub-areas of the same size, and each area is called a sensing unit.
[0036] S13. Each sensing unit has different data types in different sensing cycles, including: known data, sensed data, and reconstructed data;
[0037] As an embodiment, each sensing unit has different data types in different sensing cycles, including: data that has been sensed is called known data, sensed data (which becomes known data in the next sensing cycle), reconstructed data derived by the algorithm, and unknown data.
[0038] It should be noted that the known data in the present invention is obtained through a portable multi-parameter water quality detector or a detection pen.
[0039] S14. According to the data and data types of the sensing units, construct a data matrix DM, an environment matrix EM, a sensing matrix SM, and a reconstruction matrix RM respectively;
[0040] As an embodiment, the present invention details the data matrix DM, the environment matrix EM, the sensing matrix SM, and the reconstruction matrix RM in sequence.
[0041] Among them, data matrix (DM):
[0042] It is planned to construct an empty matrix - data matrix (DM) to store the data types represented by different sensing units in different sensing cycles. The columns of the matrix are the data types of different sensing units in the same cycle, and the rows of the matrix are the data types of a sensing unit in different cycles. Assuming that a total of n sensing units are deployed and the sensing cycle includes t time periods, then this matrix (DM) can be expressed as Figure 2 .
[0043] Environment matrix (EM):
[0044] When all the data in the data matrix are known data, an environment matrix is formed. EM indicates that each data point in the matrix is known data, that is, there are no missing data points. As Figure 3 shown, it is defined as X = (x(i,j)) n×t, where X is the environmental matrix and x(i,j) is the known data of the i-th sensing unit in the j-th sensing period.
[0045] Sensing Matrix (SM):
[0046] When part of the data matrix is sensing data and part is unknown data, a sensing matrix is formed, which is the input of the reconstruction algorithm and records the sensing data collected from the sensing personnel. Due to the existence of missing data, the elements of the SM are the collected sensing data or zero (data not collected), as Figure 4 shown, defined as S = (s(i,j)) n×t , where S is the sensing matrix and s(i,j) is the sensing data of the i-th sensing unit in the j-th sensing period.
[0047] Reconstruction Matrix (RM):
[0048] When part of the data matrix is sensing data and part is reconstructed data, a reconstruction matrix is formed, which is the output of the reconstruction algorithm, as Figure 5 shown, defined as X · = (x · (i,j)) n×t , where X· is the reconstruction matrix and x · (i,j) is the reconstructed data of the i-th sensing unit in the j-th sensing period.
[0049] S15. The urban water resource detection model based on sparse crowd-sourced sensing is specifically: given a sensing matrix SM, find an optimal reconstruction matrix RM that is as close as possible to the original data matrix EM.
[0050] As an example, after the above matrices are constructed, the problem to be solved is:
[0051] Inference of missing sensing data (VP), given a SM, find an optimal RM that is as close as possible to the original EM.
[0052] That is to say, the sum of the errors between the reconstructed data of the sensing unit and the real data is minimized:
[0053] min||X - X · || F (1)
[0054] In the VP problem, the goal is to minimize the absolute error. To measure the data reconstruction error of different methods in different scenarios, the following metrics are further defined.
[0055] Error Rate (ER): It is a measure of the error between the true sensing value and the virtual value of the reconstructed data:
[0056]
[0057] As can be seen from the above definition, in order to make the reconstructed data closer to the real data, it is necessary to minimize the value of Equation (2), that is, to find an algorithm that can reconstruct the data more accurately so that RM is closer to EM.
[0058] S2: Use the trained neural network to solve the urban water resource detection model based on sparse crowd sensing to obtain all water resource detection data in the detection area.
[0059] As an embodiment, in the present invention, a BiLSTM network is used to reconstruct the collected data to achieve full coverage detection.
[0060] The process of training the BiLSTM network is as follows:
[0061] (1) In the present invention, water quality data is collected along the 5-km lakeshore of a certain lake. This collection covers two water quality indicators: pH value and oxygen content. Using the portable multi-parameter water quality detector or detection pen mentioned in the preface, a sampling point is set every 100 m, and all sampling points are traversed and collected at intervals of 2 h. After two days of continuous detection, 4,200 pieces of water quality data are finally obtained.
[0062] (2) Process abnormal and missing data. For the collected data, the present invention uses the mean filling method to process possible missing values and uses the interquartile range method to remove outliers.
[0063] (3) Setting of hyperparameters. Epoch is the number of iterations of the training set. Selecting an appropriate number of epochs is crucial for the BiLSTM model because it directly affects the fitting effect, training efficiency, and generalization ability of the model. Setting too many epochs may lead to overfitting, while too few may lead to underfitting. Therefore, it is important to choose the right time. Figure 6 is the error rate of the model at different epochs when the data loss rate is fixed at 60%.
[0064] From Figure 6 it can be seen that as the epoch increases, the error rate gradually decreases. When the epoch is about 30, it tends to be stable. For other hyperparameters, such as window size and learning rate, the present invention uses Grid Search for automatic hyperparameter tuning.
[0065] As an embodiment, the present invention also compares the results processed by the present invention with traditional methods. Specifically as follows:
[0066] Based on the collected water quality data of a certain lake, the pH value and oxygen content in the water quality data are reconstructed using the BiLSTM algorithm and traditional data reconstruction algorithms, and a comparative experiment is conducted with the error rate as an index to measure the algorithm performance.
[0067] Based on the collected water quality data of a certain lake, the LSTM model and the SLSTM (Shuttle Long Short-Term Memory) model are trained respectively. Then, three models are used to reconstruct the pH value and dissolved oxygen in the water quality data. Taking the error rate as an index to measure the performance of the algorithm, a comparative experiment is carried out.
[0068] The first group is based on the collected water quality data of a certain lake. The BiLSTM algorithm and traditional data reconstruction algorithms are used to reconstruct the pH value and oxygen content in the water quality data. Taking the error rate as an index to measure the performance of the algorithm, the results are as Figure 7 、 8 shown.
[0069] The second group is based on the collected water quality data of a certain lake. The LSTM model and the SLSTM (Shuttle Long Short-Term Memory) model are trained respectively. Then, three models are used to reconstruct the pH value and dissolved oxygen in the water quality data. Taking the error rate as an index to measure the performance of the algorithm, the results are as Figure 9 、 10 shown.
[0070] According to the two groups of comparative experiments, as the data loss probability increases, the error rates of all models show an upward trend. However, the BiLSTM model shows better stability and lower error rate under a higher data loss probability, indicating that it has certain advantages compared with other algorithms in reconstructing water quality data.
[0071] When the data loss rate is low, the error rates of the three models are not much different. However, when the data loss rate exceeds 50%, the BiLSTM still maintains a relatively low error, while the errors of other models increase significantly, especially in the case of high data loss probability (such as 70% and 80%), indicating that the BiLSTM model has better robustness in the face of missing data in water resource detection. Therefore, using the BiLSTM algorithm for water quality data reconstruction has higher accuracy.
[0072] Please refer to Figure 11 , Figure 11 which is the schematic diagram of the working of the hardware device in the embodiment of the present invention. The hardware device specifically includes: an urban water resource detection device 401 based on sparse crowd sensing, a processor 402, and a storage medium 403.
[0073] An urban water resource detection device 401 based on sparse crowd sensing: The urban water resource detection device 401 based on sparse crowd sensing implements the urban water resource detection method based on sparse crowd sensing.
[0074] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the method for detecting urban water resources based on sparse crowd-sourced sensing.
[0075] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the method for detecting urban water resources based on sparse crowd-sourced sensing.
[0076] The beneficial effects of the present invention are as follows:
[0077] (1) High-coverage and low-cost water quality detection: With the help of portable water quality detection equipment, through the sparse crowd-sourced sensing method, comprehensive detection of a wide range of waters is achieved using a small number of collection points, overcoming the problems of low coverage and high cost of traditional methods.
[0078] (2) Improvement in data reconstruction accuracy: The bidirectional long short-term memory network (BiLSTM) is used to reconstruct the incomplete water quality data collected. Compared with traditional reconstruction algorithms and other deep learning models, BiLSTM shows higher accuracy and stability, especially in the case of a high data loss rate.
[0079] (3) Ability to handle irregular sensing areas: In response to the problems of complex shapes of water area detection areas and limited collection conditions, the proposed model can handle the irregularity of the sensing area, and the reconstruction results meet the requirements of the environmental protection department for water quality detection accuracy.
[0080] (4) Noise resistance and robustness: In the face of a high data loss rate and noise in the sensed data, the model can effectively filter out the noise and compensate for the missing data, ensuring the reliability and stability of the detection results.
[0081] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.
Claims
1. A method for urban water resources detection based on sparse crowd intelligence perception, characterized by: The method comprises the following steps: S1: Construct an urban water resources detection model based on sparse crowd sensing; S2: Use the trained neural network to solve the urban water resource detection model based on sparse crowd intelligence perception to obtain all water resource detection data in the detection area.
2. The urban water resources detection method based on sparse crowd intelligence perception according to claim 1 is characterized by: Step S1 is specifically as follows: S11. Study the water pollution investigation reports of various cities and identify the pollution sources; S12, taking the pollution source area as the sensing area, and dividing the sensing area into a plurality of sensing units of the same size; S13, each sensing unit has different data types in different sensing cycles, including: known data, sensed data and reconstructed data; S14, constructing a data matrix DM, an environment matrix EM, a perception matrix SM, and a reconstruction matrix RM respectively according to the data and data types of the perception unit; S15. The urban water resource detection model based on sparse crowd intelligence perception is specifically as follows: given a perception matrix SM, an optimal reconstruction matrix RM is found that is as close as possible to the original data matrix EM.
3. The urban water resources detection method based on sparse crowd intelligence perception as claimed in claim 2 is characterized by: In step S14, the data matrix DM is used to store the data types represented by different sensing units in different sensing periods. The columns of the matrix are the data types of different sensing units in the same period, and the rows of the matrix are the data types of a sensing unit in different periods.
4. The urban water resources detection method based on sparse crowd intelligence perception as claimed in claim 2, characterized in that: In step S14, the environment matrix EM is composed of known data.
5. The urban water resources detection method based on sparse crowd intelligence perception as claimed in claim 2, characterized in that: In step S14, the sensing matrix SM is composed of part of the sensing data and part of the unknown data.
6. The urban water resources detection method based on sparse crowd intelligence perception according to claim 2, characterized in that: The reconstruction matrix RM is composed of part of the sensed data and part of the reconstructed data.
7. The urban water resources detection method based on sparse crowd intelligence perception according to claim 2 is characterized by: In step S2, the neural network adopts a BiLSTM network.
8. The urban water resources detection method based on sparse crowd intelligence perception according to claim 7 is characterized by: Step S2 specifically refers to inputting the given perception matrix SM into the trained BiLSTM network to obtain the reconstruction matrix RM as the detection data of all water resources in the detection area.
9. A storage medium, characterized in that: The storage medium stores instructions and data for implementing a method for detecting urban water resources based on sparse crowd intelligence perception as described in any one of claims 1 to 8.
10. An urban water resources detection device based on sparse group intelligence perception, characterized by: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement the urban water resources detection method based on sparse crowd intelligence perception as described in any one of claims 1 to 8.