Data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a data processing method and device, electronic equipment and a storage medium. The method comprises the following steps: iteratively dividing a target water quality index sequence of a water body into a first data set and a second data set along a time axis; ranking the first data set from small to large, and taking a numerical value in a first set rank as a first distribution characteristic value; ranking the second data set from small to large, and taking a numerical value in a second set rank as a second distribution characteristic value; and when the jump intensity of the second distribution characteristic value relative to the first distribution characteristic value and the data volume of the first data set meet a jump judgment condition, generating a jump alarm signal containing a jump intensity parameter. When the water quality is monitored, data jump identification is carried out, and the accuracy and the reliability of a monitoring result are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] As a fundamental natural resource for maintaining ecosystem balance and human societal development, the quality and safety of water resources are directly related to public health and economic sustainability. Against the backdrop of rapid urbanization and intensive industrial and agricultural development, the incidence of abnormal water quality events in key river basins in my country is continuously increasing.

[0003] In the current field of online water quality monitoring (such as automatic river section stations and sewage treatment plant outlet monitoring), the parameters such as pH value, dissolved oxygen, and turbidity collected by sensors often exhibit abnormal jumps in data due to equipment status or environmental interference (not caused by water pollution itself). This data can be confused with the numerical changes caused by actual pollutant discharge, resulting in a decrease in the accuracy of alarm information. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a data processing method, apparatus, electronic device and storage medium to overcome the problems in the prior art.

[0005] In a first aspect, embodiments of this application provide a data processing method, the method comprising: The target water quality index sequence of the water body is iteratively divided along the time axis into the first dataset and the second dataset; The first dataset is sorted from smallest to largest, and the values ​​in the first predetermined ranking are used as the first distribution feature values; and the second dataset is sorted from smallest to largest, and the values ​​in the second predetermined ranking are used as the second distribution feature values. When the jump intensity of the second distribution feature value relative to the first distribution feature value and the data volume of the first dataset meet the jump judgment condition, a jump alarm signal containing a jump intensity parameter is generated.

[0006] Some technical solutions and methods in this application obtain the target water quality index sequence in the following ways: Acquire all raw water quality index sequences collected by the acquisition equipment; Based on the conductivity in the original water quality index sequence, and based on the preset first operating condition characteristic index, abnormal operating condition data are removed to obtain the candidate water quality index sequence after removal. The abnormal operating condition data are the water quality indexes in the original water quality index sequence corresponding to the time when the conductivity does not meet the first operating condition characteristic index. The target water quality indicator sequence is selected from the candidate water quality indicator sequences based on the required data volume. And / or, based on the water level when the original water quality index sequence is collected, and based on the preset second operating condition characteristic index, abnormal operating condition data are removed to obtain the candidate water quality index sequence after removal, wherein the abnormal operating condition data are: water quality indexes in the original water quality index sequence corresponding to the time when the water level does not meet the second operating condition characteristic index. The target water quality index sequence is selected from the candidate water quality index sequences based on the required data volume.

[0007] In some technical solutions of this application, the target water quality indicator sequence is selected from the candidate water quality indicator sequences based on the data demand, including: Based on the required data volume, a preliminary water quality indicator sequence is selected from the candidate water quality indicator sequences. By performing aggregation processing on the initially selected water quality index sequence, a target water quality index sequence with dimensionality reduction characteristics is generated.

[0008] In some technical solutions of this application, the above-mentioned jump judgment conditions include a first judgment condition and a second judgment condition. The step of generating a jump alarm signal containing a jump intensity parameter when the jump intensity of the second distribution feature value relative to the first distribution feature value and the data volume of the first dataset satisfy the jump judgment condition includes: When the jump intensity of the second distribution feature value relative to the first distribution feature value satisfies the first judgment condition and the data volume of the first dataset satisfies the second judgment condition, a jump alarm signal containing a jump intensity parameter is generated.

[0009] In some technical solutions of this application, the above method determines the jump strength of the second distribution characteristic value relative to the first distribution characteristic value in the following way: Calculate the difference between the values ​​corresponding to the second set ranking and the first set ranking, and calculate the jump ratio of the difference to the value of the first set ranking. The determination is made based on the jump percentage and a preset percentage threshold; and / or, The method determines the amount of data in the first dataset in the following way: The judgment is made by comparing the amount of data in the first dataset with the preset upper and lower thresholds for the number of data.

[0010] In some technical solutions of this application, the aforementioned transition alarm signal includes a transition time identifier, and the method further includes: Obtain all alarm signals within the time period corresponding to the jump time identifier; The transition alarm signal is removed from all the alarm signals to obtain the optimized alarm signal, and an alarm is issued based on the optimized alarm signal.

[0011] In some technical solutions of this application, the above method also includes: identifying the jump intensity parameter and determining the alarm type corresponding to the jump alarm signal.

[0012] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising: The partitioning module is used to iteratively partition the target water quality index sequence of the water body into a first dataset and a second dataset along the time axis. The sorting module is used to sort the first dataset in ascending order, with the values ​​in the first set rank as the first distribution feature values; and to sort the second dataset in ascending order, with the values ​​in the second set rank as the second distribution feature values. The judgment module is used to generate a jump alarm signal containing a jump intensity parameter when the jump intensity of the second distribution feature value relative to the first distribution feature value and the data volume of the first dataset meet the jump judgment condition.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the data processing method described above.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data processing method described above.

[0015] The technical solutions provided by the embodiments of this application may include the following beneficial effects: The method of this application includes: iteratively dividing the target water quality index sequence of a water body into a first dataset and a second dataset along a time axis; ranking the first dataset from smallest to largest, with the value at a first predetermined rank as a first distribution feature value; and ranking the second dataset from smallest to largest, with the value at a second predetermined rank as a second distribution feature value; when the jump intensity of the second distribution feature value relative to the first distribution feature value and the data volume of the first dataset meet the jump judgment condition, a jump alarm signal including a jump intensity parameter is generated. This application identifies data jumps during water quality monitoring, improving the accuracy and reliability of the monitoring results.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application is shown. Figure 2 A schematic diagram of a transition alarm signal provided in an embodiment of this application is shown; Figure 3 A schematic diagram of a data processing apparatus provided in an embodiment of this application is shown; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0020] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0022] Water resources are essential for human survival, and the quality of drinking water is directly related to human life and safety. With decades of rapid economic development, environmental pollution incidents have also become increasingly frequent, and recent major water pollution incidents have caused severe social, economic, and environmental losses. Therefore, water quality monitoring is necessary.

[0023] Current technology issues alarms when abnormal data is detected in water quality monitoring, without considering the reasons for data jumps within the abnormal data. Data jumps not caused by water quality issues lead to false alarms, resulting in inaccurate alarms and wasted resources in alarm processing.

[0024] Based on this, embodiments of this application provide a data processing method, apparatus, electronic device, and storage medium for identifying data jumps not caused by water quality (pollution, etc.) itself (e.g., caused by equipment malfunction, operating condition malfunction, etc.), which are described below through embodiments.

[0025] Figure 1 This illustration shows a flowchart of a data processing method provided in an embodiment of this application, which includes steps S101-S104; specifically: S101. The target water quality index sequence for dividing the water body iteratively along the time axis is the first dataset and the second dataset. S102. The first dataset is sorted from smallest to largest, and the values ​​in the first set ranking are used as the first distribution feature values; and the second dataset is sorted from smallest to largest, and the values ​​in the second set ranking are used as the second distribution feature values. S103. When the jump intensity of the second distribution feature value relative to the first distribution feature value and the data volume of the first dataset meet the jump judgment condition, a jump alarm signal containing a jump intensity parameter is generated. The jump intensity parameter includes: jump time and jump ratio, and these data will be recorded and stored.

[0026] This application identifies data jumps during water quality monitoring, thereby improving the accuracy and reliability of the monitoring results.

[0027] The following describes some embodiments of this application in detail. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0028] This application provides a data processing method, which can be executed by an electronic device such as a terminal device or a server. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. The method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, the method can be executed by a server.

[0029] This data processing method is applied to water body monitoring, where water body is a collection of water, such as rivers, lakes, seas, reservoirs, groundwater, etc. Water body includes not only water, but also solvents, suspended solids, bottom sediment, aquatic organisms, etc. The embodiments of this application do not limit the types of water bodies. When monitoring water quality, it is necessary to obtain the target water quality index sequence of the water body. In specific implementation, a quantum dot spectrometer can be used to measure the water quality information. The quantum dot spectrometer may include a quantum dot spectral probe, which measures incident light (e.g., light transmitted or scattered after passing through a predetermined area of ​​water sample) based on the physical and optical properties of nanocrystals to obtain the spectral information of the incident light. For example, the quantum dot spectral probe may include a nanocrystal chip made of various nanocrystals, wherein the nanocrystal chip contains a certain arrangement of various nanocrystals (e.g., a nanocrystal array), wherein each nanocrystal has different light absorption or emission characteristics, and different types of semiconductor nanocrystals. For example, they can be made of different materials and sizes, so that the nanocrystal chip can modulate the response of wavelengths over a wide wavelength range to obtain a spectrum adjusted for incident light over a wide wavelength range.

[0030] Light transmitted or scattered through water can be affected by substances in the water (e.g., suspended solids, pollutants), thus obtaining specific spectral information. A quantum dot spectrometer can acquire this spectral information in real time, which can represent the water quality information of the water body at the measurement location, and further determine water quality indicators based on this spectral information. For example, by observing the absorption intensity of different wavelengths of light by a water sample, spectral information of different frequency bands of light can be obtained, and water quality indicators can be calculated from this spectral information. This application does not limit the working principle of the quantum dot spectrometer.

[0031] Water quality indicators may include Chemical Oxygen Demand (COD), turbidity, total phosphorus content, ammonia nitrogen content, permanganate index, total suspended solids, biological oxygen demand, total organic carbon, sulfate content, chloride content, dissolved iron content, dissolved manganese content, dissolved copper content, dissolved zinc content, nitrate content, nitrite content, total nitrogen content, fluoride content, selenium content, total arsenic content, total mercury content, total cadmium content, chromium content, total lead content, total cyanide, volatile phenol content, coliform bacteria content, and sulfide content. Water temperature can also be determined using infrared spectroscopy from the spectral information.

[0032] In the example, a quantum dot spectrometer can determine water quality indicators by analyzing the light absorption characteristics of various substances contained in water. For example, it can analyze the light intensity of light at a specific wavelength by using spectral information, thereby obtaining the concentration (water quality indicator) of the substance corresponding to the light within that specific wavelength range. Alternatively, the quantum dot spectrometer can infer water quality indicators using a neural network. For example, spectral information can be input into the neural network, which can then infer the concentration (water quality indicator) of various substances. This application does not limit the method of determining water quality indicators.

[0033] Quantum dot spectral probes can be used to measure spectral data in real time, and water quality indicators (e.g., COD) can be calculated based on the spectral data. Water quality indicators obtained at multiple times at the same location can be combined to form a raw water quality indicator sequence. Compared to the method of sampling water and then conducting laboratory tests to measure water quality indicators, measuring indicators using quantum dot spectral probes can achieve online, in-situ, high-frequency, and real-time measurement. For example, the measurement frequency can be increased from once a day to 3-60 minutes per measurement, preferably 5-30 minutes per measurement, particularly preferably 8-20 minutes per measurement, and most preferably 10-15 minutes per measurement, which is much higher than traditional testing methods. Therefore, water quality indicator sequences can be obtained at a higher frequency.

[0034] Water quality indicators (e.g., COD) of water bodies can be continuously monitored using a quantum dot spectrometer at a preset location to obtain a raw water quality indicator sequence. In the example, the raw water quality indicator sequence can be represented as: {(t1,x1,1),(t2,x1,2),(t3,x1,3),…,(tc,x1,c),…}, where c is any positive integer, tc represents the c-th time, and x1,c represents the water quality indicator measured at the c-th time.

[0035] Even when the water level changes (below the level of the quantum dot spectrometer), the quantum dot spectrometer is still acquiring data, and will still collect data from outside the water. When monitoring water bodies, it is necessary to exclude this data. Considering the differences between data from outside the water and data from inside the water (data collected by the quantum dot spectrometer in the water), conductivity and water level are the most prominent factors. Preferably, in this embodiment, data from outside the water is excluded based on conductivity and / or based on the water level at the time the original water quality index sequence was collected.

[0036] Specifically, the conductivity of the original water quality index sequence is determined. Based on the preset first operating condition characteristic index, abnormal operating condition data is removed to obtain the candidate water quality index sequence after removal. The abnormal operating condition data refers to the water quality index in the original water quality index sequence corresponding to the moment when the conductivity does not meet the first operating condition characteristic index. For example, abnormal water-free data with conductivity less than a preset conductivity threshold (such as <1μS / cm) in the original water quality index sequence are removed to obtain the candidate water quality index sequence after removal.

[0037] And / or, obtain the water level when the original water quality index sequence is collected, and based on the preset second working condition characteristic index, remove abnormal working condition data to obtain the candidate water quality index sequence after removal, wherein the abnormal working condition data are: the water quality index in the original water quality index sequence corresponding to the time when the water level does not meet the second working condition characteristic index.

[0038] In actual working scenarios, the number of candidate water quality index sequences is quite large. If all candidate water quality index sequences are processed, it will consume a lot of computing resources. In order to improve efficiency, after obtaining the candidate water quality index sequences, this application embodiment needs to further screen the candidate water quality index sequences to obtain the screened target water quality index sequences.

[0039] When screening the water quality indicator sequences to be selected, the primary basis is the data requirement in the actual working scenario. In other words, this embodiment of the application needs to select a required number of water quality indicator sequences from the candidate sequences as the initial selection. In specific implementation, the required data quantity can be limited by a preset collection time period, or directly by the number of data points. For example, three days' worth of water quality indicator sequences can be selected as the initial selection, or 400 consecutive water quality indicator sequences can be selected. Of course, other limiting methods are not excluded in this embodiment of the application.

[0040] While the above method yields the required number of preliminary water quality indicator sequences for practical work scenarios, the probability of data jumps is low, or the changes in water quality indicators are small. Therefore, if each preliminary water quality indicator sequence is tested separately, the detection effect is still relatively low. Based on this, this embodiment does not test the preliminary water quality indicator sequences, but instead performs aggregation processing on them to obtain aggregated target water quality indicator sequences, which are then tested. This aggregation processing can include calculating the median, average, etc., and the specific aggregation processing method is not limited.

[0041] After obtaining the target water quality index sequence through the above method, in order to determine the data jumps, this embodiment of the application also records the collection time of each target water quality index sequence and then traverses the target water quality index sequence in the order of collection time. Specifically, this embodiment of the application iteratively divides the target water quality index sequence to obtain a first dataset (already traversed) and a second dataset (not yet traversed). The iterative division includes dividing the target water quality index sequence into a first dataset and a second dataset. The first dataset contains M consecutive index values ​​starting from the first water quality index value, and the second dataset contains the subsequent NM index values, where M is a positive integer increasing from 1 to N-1, and N is the total number of index values ​​in the target water quality index sequence.

[0042] For example, if the target water quality index sequence is A1, A2, A3, A4, A5, A6, A7, A8, A9, etc., the target water quality index sequence is divided into two datasets: A1 (first dataset) and A2-A9 (second dataset). Then, it is further divided into two datasets: A1, A2, and A3-A9. This process is repeated until a jump in data is identified.

[0043] The specific detection process in this application embodiment is as follows: calculate the first distribution feature value of the first dataset and the second distribution feature value of the second dataset, and then make a judgment based on the first distribution feature value and the second distribution feature value.

[0044] In statistics, data characteristics can be categorized into three main types: central tendency, dispersion, and distribution shape, which respectively describe the central location, dispersion, and shape of the data. Central tendency characteristics reflect the core location and typicality of the dataset. Common examples include the mode and median. Dispersion characteristics reflect the range and dispersion of data, mainly including the range, variance, and standard deviation. Distribution shape characteristics reveal the symmetry and kurtosis of the data distribution, commonly including skewness and kurtosis. Preferably, this application embodiment uses distribution shape type to characterize the characteristics of the first and second datasets. More preferably, this application embodiment uses percentiles from the distribution shape type to characterize the characteristics of the first and second datasets.

[0045] In specific implementation, this application embodiment calculates the values ​​of the first predetermined rank of the first dataset and the second predetermined rank of the second dataset. Then, a data jump judgment is made based on the values ​​of the first and second predetermined rank. Preferably, the predetermined rank here uses percentiles. Percentiles are used to describe the distribution characteristics of a set of quantitative data. It indicates what percentage of data values ​​in a set of data are less than or equal to a certain value. For example, the pth percentile means that in a set of observations arranged in ascending order, p% of the data values ​​are less than or equal to this number, while (100-p)% of the data values ​​are greater than or equal to this number. For ungrouped data, a method of directly looking up the data after sorting is usually used. For example, to find the 50th percentile (i.e., the median), simply sort the data from smallest to largest and take the middle number. For grouped data, interpolation can be used to calculate percentiles. The specific method involves determining the position of the target percentile based on the frequency and cumulative frequency of each group, and performing interpolation calculation using the boundary values ​​and frequencies of adjacent groups.

[0046] For example, calculate the 90th percentile of the first dataset and the 10th percentile of the second dataset; calculate the 80th percentile of the first dataset and the 20th percentile of the second dataset; calculate the 70th percentile of the first dataset and the 30th percentile of the second dataset, etc.

[0047] After obtaining the values ​​of the first predetermined rank of the first dataset and the second predetermined rank of the second dataset, this embodiment of the application needs to determine whether a jump has occurred in the target water quality indicator sequence based on the values ​​of the first predetermined rank of the first dataset and the second predetermined rank of the second dataset. Since this embodiment aims to determine data jumps, for accuracy, the judgment basis here is not the values ​​of the first and second predetermined rank, but rather the calculated jump intensity of the second distribution characteristic value relative to the first distribution characteristic value. The specific jump intensity can be represented by (MN) / N. For example, the jump intensity of the 90th percentile (p90) of the first dataset and the 10th percentile (p10) of the second dataset is (p10-p90) / p90.

[0048] In practical work scenarios, if the amount of data in the first dataset is small, it may not be sufficient to establish a stable statistical basis. For example, if only a 2-hour target water quality index series is used to calculate the P90 (90th percentile), the insufficient data may lead to a large deviation in quantile estimation, resulting in a misjudgment of a data jump. Considering the sufficiency of data, this application embodiment, in addition to considering the jump strength of the second distribution characteristic value relative to the first distribution characteristic value, also needs to consider the amount of data in the first dataset. That is, this application embodiment comprehensively judges the jump strength of the second distribution characteristic value relative to the first distribution characteristic value and the amount of data in the first dataset to determine whether a data jump has occurred.

[0049] Preferably, when comprehensively considering the jump intensity of the second distribution feature value relative to the first distribution feature value and the first dataset, since the two have different focuses, the judgment criteria for the two are further refined in this embodiment: a first judgment condition is used to judge the jump intensity of the second distribution feature value relative to the first distribution feature value; a second judgment condition is used to judge the data volume of the first dataset. That is, when the jump intensity of the second distribution feature value relative to the first distribution feature value meets the first judgment condition and the data volume of the first dataset meets the second judgment condition, this embodiment considers that the target water quality indicator sequence has jumped and generates a jump alarm signal.

[0050] For example, the first judgment condition may conform to a preset sliding window trend, or conform to the slope feature after filtering and decomposition, or be greater than a preset jump threshold, etc. Considering the above embodiments, this application embodiment uses the jump intensity method of calculating the jump ratio of the difference between the value of the second set ranking and the value of the first set ranking in the jump ratio of the first set ranking. Preferably, this application embodiment uses exceeding a preset ratio threshold as the first judgment condition. The second judgment condition is used to limit the data volume of the first dataset. In specific implementation, the number of the first dataset can be counted to limit the data volume of the first dataset to the range between the upper limit threshold and the lower limit threshold. On the other hand, when monitoring water quality, since the data collection frequency is fixed after being set, the data volume of the first dataset can also be limited to the range between the upper limit threshold and the lower limit threshold by restricting the time interval of the first dataset collection.

[0051] The aforementioned percentage threshold can be obtained based on historical data and a preset model, that is, the percentage threshold is calculated based on the data that shows a jump in historical data.

[0052] For example, such as Figure 2 The red vertical line indicates that when traversing to 05:00:00 on day2, the following conditions are met: (p10-p90) / p90>0.5, and the collection time of the first dataset is greater than the first preset duration (24 hours) and less than or equal to the second preset duration (48 hours). The jump ratio ((p10-p90) / p90) is recorded, and a jump alarm is issued.

[0053] To facilitate impact handling by staff, this embodiment of the application, after generating a jump alarm signal, further identifies the jump alarm signal and determines the corresponding alarm type. The identification of the jump alarm signal is based on the jump intensity parameter contained within it, which is a specific value calculated based on the jump intensity. Specifically, the identification can be performed by comparing the jump intensity parameter with a pre-stored system disturbance feature library, or by using a pre-trained prediction model to predict and identify the jump intensity parameter.

[0054] For example, the preset system disturbance feature library includes multiple different intensity ranges (a first intensity range, a second intensity range, and a third intensity range). Different intensity ranges correspond to different alarm types (the first intensity range corresponds to system disturbances caused by sensor physical contamination; the second intensity range corresponds to system disturbances caused by circuit signal interference; and the third intensity range corresponds to system disturbances caused by sensor hardware failure). By comparing the jump intensity parameter with the system disturbance feature library, if the jump intensity parameter is determined to be within the second intensity range, then the alarm type indicating a data jump is determined to be a system disturbance caused by circuit signal interference.

[0055] This application embodiment identifies the target water quality indicator sequence using the above method, enabling high-precision determination of abrupt changes in the target water quality indicator sequence. Furthermore, it identifies the specific alarm type of the abrupt change, facilitating rapid repairs and ensuring work efficiency.

[0056] In an optional implementation, after identifying the target water quality indicator sequence, this application embodiment can also provide feedback on the water quality monitoring process. Specifically, it can eliminate alarms triggered by data jumps during water quality monitoring, increasing the proportion of alarms caused by water body anomalies. Specifically, the jump alarm signal generated in this application embodiment also includes a corresponding jump time identifier. Since the target water quality indicator sequence in this application embodiment has undergone aggregation processing, the jump time identifier here is a time period. For example, the time identifier could be 9:00~9:20, etc. During water body monitoring, this time identifier may trigger multiple alarm signals. These alarm signals include: jump alarm signals (abnormal alarm signals, excluding jump alarm signals caused by water quality anomalies) and water quality anomaly alarm signals. After identifying the jump alarm signal, this application embodiment can eliminate it to avoid false alarms. Alternatively, it can first remove the data corresponding to the jump alarm signal from the water quality indicator sequence, and then perform alarm judgment based on the water quality data after removing the jump data and preset alarm rules, improving alarm accuracy.

[0057] Figure 3 This invention provides a schematic diagram of the structure of a data processing apparatus according to an embodiment of the present application. The apparatus includes: The partitioning module is used to iteratively partition the target water quality index sequence of the water body into a first dataset and a second dataset along the time axis. The sorting module is used to sort the first dataset in ascending order, with the values ​​in the first set rank as the first distribution feature values; and to sort the second dataset in ascending order, with the values ​​in the second set rank as the second distribution feature values. The judgment module is used to generate a jump alarm signal containing a jump intensity parameter when the jump intensity of the second distribution feature value relative to the first distribution feature value and the data volume of the first dataset meet the jump judgment condition.

[0058] The segmentation module includes an acquisition module, which is used to acquire all the original water quality index sequences collected by the acquisition equipment; Based on the conductivity in the original water quality index sequence, and based on the preset first operating condition characteristic index, abnormal operating condition data are removed to obtain the candidate water quality index sequence after removal. The abnormal operating condition data are the water quality indexes in the original water quality index sequence corresponding to the time when the conductivity does not meet the first operating condition characteristic index. The target water quality indicator sequence is selected from the candidate water quality indicator sequences based on the required data volume. And / or, based on the water level when the original water quality index sequence is collected, and based on the preset second operating condition characteristic index, abnormal operating condition data are removed to obtain the candidate water quality index sequence after removal, wherein the abnormal operating condition data are: water quality indexes in the original water quality index sequence corresponding to the time when the water level does not meet the second operating condition characteristic index. The target water quality index sequence is selected from the candidate water quality index sequences based on the required data volume.

[0059] The acquisition module, used to filter the target water quality indicator sequence from the candidate water quality indicator sequences according to the required data volume, includes: Based on the required data volume, a preliminary water quality indicator sequence is selected from the candidate water quality indicator sequences. By performing aggregation processing on the initially selected water quality index sequence, a target water quality index sequence with dimensionality reduction characteristics is generated.

[0060] The jump judgment condition includes a first judgment condition and a second judgment condition. The judgment module, when generating a jump alarm signal containing a jump intensity parameter when the jump intensity of the second distribution feature value relative to the first distribution feature value and the data volume of the first dataset satisfy the jump judgment condition, includes: When the jump intensity of the second distribution feature value relative to the first distribution feature value satisfies the first judgment condition and the data volume of the first dataset satisfies the second judgment condition, a jump alarm signal containing a jump intensity parameter is generated.

[0061] The judgment module determines the abrupt change intensity of the second distribution feature value relative to the first distribution feature value in the following manner: Calculate the difference between the values ​​corresponding to the second set ranking and the first set ranking, and calculate the jump ratio of the difference to the value of the first set ranking. The judgment is made based on the jump ratio and the preset ratio threshold; The method determines the amount of data in the first dataset in the following way: The judgment is made by comparing the amount of data in the first dataset with the preset upper and lower thresholds for the number of data.

[0062] The transition alarm signal includes a transition time identifier. The device further includes: a processing module, used to acquire all alarm signals within the time period corresponding to the jump time identifier; The jump intensity parameter is identified to determine the alarm type corresponding to the jump alarm signal.

[0063] like Figure 4 As shown, this application provides an electronic device for executing the data processing method described in this application. The device includes a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the data processing method described above.

[0064] Specifically, the aforementioned memory and processor can be general-purpose memory and processor, without any specific limitations. When the processor runs the computer program stored in the memory, it can execute the aforementioned data processing methods.

[0065] Corresponding to the data processing method in this application, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the data processing method described above.

[0066] Specifically, the storage medium can be a general-purpose storage medium, such as a removable disk or hard disk, and when the computer program on the storage medium is run, it can perform the above-mentioned data processing methods.

[0067] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the coupling or direct coupling or communication connection shown or discussed may be through some communication interface; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0068] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0069] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0070] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0072] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: The target water quality index sequence of the water body is iteratively divided along the time axis into the first dataset and the second dataset; The first dataset is sorted from smallest to largest, and the values ​​in the first predetermined ranking are used as the first distribution feature values; and the second dataset is sorted from smallest to largest, and the values ​​in the second predetermined ranking are used as the second distribution feature values. When the jump intensity of the second distribution feature value relative to the first distribution feature value and the data volume of the first dataset meet the jump judgment condition, a jump alarm signal containing a jump intensity parameter is generated.

2. The method according to claim 1, characterized in that, The method obtains the target water quality index sequence in the following manner: Acquire all raw water quality index sequences collected by the acquisition equipment; Based on the conductivity in the original water quality index sequence, and based on the preset first operating condition characteristic index, abnormal operating condition data are removed to obtain the candidate water quality index sequence after removal. The abnormal operating condition data are the water quality indexes in the original water quality index sequence corresponding to the time when the conductivity does not meet the first operating condition characteristic index. The target water quality indicator sequence is selected from the candidate water quality indicator sequences based on the required data volume. And / or, based on the water level when the original water quality index sequence is collected, and based on the preset second operating condition characteristic index, abnormal operating condition data are removed to obtain the candidate water quality index sequence after removal, wherein the abnormal operating condition data are: water quality indexes in the original water quality index sequence corresponding to the time when the water level does not meet the second operating condition characteristic index. The target water quality index sequence is selected from the candidate water quality index sequences based on the required data volume.

3. The method according to claim 2, characterized in that, The step of selecting the target water quality indicator sequence from the candidate water quality indicator sequences based on the data demand includes: Based on the required data volume, a preliminary water quality indicator sequence is selected from the candidate water quality indicator sequences. By performing aggregation processing on the initially selected water quality index sequence, a target water quality index sequence with dimensionality reduction characteristics is generated.

4. The method according to claim 1, characterized in that, The jump judgment condition includes a first judgment condition and a second judgment condition. When the jump intensity of the second distribution feature value relative to the first distribution feature value and the data volume of the first dataset satisfy the jump judgment condition, a jump alarm signal containing a jump intensity parameter is generated, including: When the jump intensity of the second distribution feature value relative to the first distribution feature value satisfies the first judgment condition and the data volume of the first dataset satisfies the second judgment condition, a jump alarm signal containing a jump intensity parameter is generated.

5. The method according to claim 4, characterized in that, The method determines the abrupt change strength of the second distribution feature value relative to the first distribution feature value in the following manner: Calculate the difference between the values ​​corresponding to the second set ranking and the first set ranking, and calculate the jump ratio of the difference to the value of the first set ranking. The determination is made based on the jump percentage and a preset percentage threshold; and / or, The method determines the amount of data in the first dataset in the following way: The judgment is made by comparing the amount of data in the first dataset with the preset upper and lower thresholds for the number of data.

6. The method according to any one of claims 1 to 5, characterized in that, The transition alarm signal includes a transition time identifier, and the method further includes: Obtain all alarm signals within the time period corresponding to the jump time identifier; The transition alarm signal is removed from all the alarm signals to obtain the optimized alarm signal, and an alarm is issued based on the optimized alarm signal.

7. The method according to any one of claims 1 to 5, characterized in that, The method further includes: identifying the jump intensity parameter and determining the alarm type corresponding to the jump alarm signal.

8. A data processing apparatus, characterized in that, The device includes: The partitioning module is used to iteratively partition the target water quality index sequence of the water body into a first dataset and a second dataset along the time axis. The sorting module is used to sort the first dataset in ascending order, with the values ​​in the first set rank as the first distribution feature values; and to sort the second dataset in ascending order, with the values ​​in the second set rank as the second distribution feature values. The judgment module is used to generate a jump alarm signal containing a jump intensity parameter when the jump intensity of the second distribution feature value relative to the first distribution feature value and the data volume of the first dataset meet the jump judgment condition.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the memory via the bus, and the machine-readable instructions, when executed by the processor, perform the steps of the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data processing method as described in any one of claims 1 to 7.

Images