Data processing method of multi-type sensing equipment based on field monitoring system

Abstract

The invention relates to a data processing method for multi-type sensing equipment based on a field monitoring system. The method comprises the following steps: acquiring data acquired by each type of sensing equipment; for each type, determining a reference value of each sensing device of the type according to a mean value and a standard deviation of data collected by each sensing device of the type; according to the reference value, determining whether abnormal data exists in the data collected by the sensing devices of the type; labeling the collected data based on the abnormal data; and performing data processing based on the labeled data. According to the method, after the data are collected, the data are not directly processed, but after the abnormal data exist in the collected data and the abnormal data are labeled, data processing is carried out based on the labeled data, so that the abnormal data can be focused on according to the labeling of the data during data processing, and the accuracy of data processing is improved. And the accuracy of a processing result is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data processing method for multi-type sensor equipment based on a field monitoring system. Background Art

[0002] With the development of computer and communication technology, field monitoring systems have been more widely used.

[0003] At present, the field monitoring system is composed of hardware equipment such as sensor equipment and monitoring host. It forms an intelligent monitoring network through wireless communication network to achieve 24-hour uninterrupted field environment monitoring. It has the advantages of not being affected by geographical environment, climate, time and other factors.

[0004] At present, the data of various types of sensor equipment in the field detection system will be processed directly. In the field monitoring process, the data volume is large and most of the data does not change much. For example, when detecting wild animals, sensor equipment will be installed at the location where the wild animals are found, and the sensor equipment will continuously collect data, and then the wild animals will be found from the data. However, the whereabouts of wild animals are uncertain. Most of the time, wild animals will not appear in the data collection area. At this time, most of the data collected by the sensor equipment is data in the absence of wild animals (this application refers to normal data), and the data with wild animals (this application refers to abnormal data) is very small. If the collected data is processed directly, the data with wild animals will be submerged in the data without wild animals and will be difficult to be found, thereby affecting the accuracy of the processing results. Summary of the Invention

[0005] (1) Technical issues to be solved

[0006] In response to the existing technical problems, the present invention provides a data processing method for multiple types of sensor equipment based on a field monitoring system.

[0007] (2) Technical solution

[0008] In order to achieve the above objectives, the main technical solutions adopted by the present invention include:

[0009] A data processing method for multiple types of sensor equipment based on a field monitoring system, the method comprising:

[0010] Obtain data collected by various types of sensor devices;

[0011] For each type, a baseline value of each sensor device of the type is determined based on the mean and standard deviation of the data collected by each sensor device of the type; and whether the data collected by each sensor device of the type contains abnormal data is determined based on the baseline value;

[0012] Annotate the collected data based on abnormal data;

[0013] Perform data processing based on the labeled data.

[0014] Optionally, for any type i, the baseline value of each sensor device of the type is determined based on the mean and standard deviation of the data collected by each sensor device of the type, including:

[0015] For any sensor device j of any type i, the baseline value is determined by the following steps:

[0016] Determine the mean of the data collected by all sensor devices of type i Standard deviation majority

[0017] Determine the mean of the data collected by sensor device j Standard deviation majority

[0018] Determine the relationship that the sensing device j satisfies; where and The relationship between and If the relationship between them is the same, it is determined that the sensor device j satisfies the first type of relationship; otherwise, it is determined that the sensor device j satisfies the second type of relationship;

[0019] Sort the data collected by sensor device j from recent to farthest in time to obtain the collected data sequence of sensor device j;

[0020] Divide the data sequence collected by sensor device j into segments according to the preset time length and determine the mean of each segment Standard deviation majority Among them, u j is the segment identifier;

[0021] According to the relationship satisfied by the sensing device j and the Determine the adjustment coefficient for sensing device j

[0022] According to the relationship satisfied by the sensor device j and Determine the baseline value of sensing device j

[0023] Optionally, according to the relationship satisfied by the sensing device j and the Determine the adjustment factor for the sensing device h include:

[0024] If the sensor device j satisfies the first type of relationship, then according to the Determine the adjustment coefficient for sensing device j

[0025] If the sensor device j satisfies the second type of relationship, the standard deviation of the data collected by each sensor device is used to determine

[0026] Optionally, according to the relationship satisfied by the sensing device j and Determine the baseline value of sensing device j include:

[0027] If the sensor device j satisfies the first type of relationship, then according to and each section Determine the baseline value of sensing device j

[0028] If the sensor device j satisfies the second type of relationship, then according to and Determine the baseline value of sensing device j

[0029] Optionally, for any type i, determining whether the data collected by each sensor device of the type contains abnormal data based on the reference value includes:

[0030] For any sensor device j of any type i, the data collected by it that does not fall within the normal range is determined as abnormal data;

[0031] The normal interval is determined based on the reference value of the sensor device j and the standard deviation of the data collected by the sensor device j.

[0032] Optionally, the collected data is labeled based on the abnormal data, including:

[0033] Record the collection time of abnormal data of each type of sensor equipment;

[0034] Determine the abnormal time interval of each type of sensor device according to the collection time of abnormal data of each type of sensor device;

[0035] Determine the labeled data based on the abnormal time interval of each type of sensor equipment;

[0036] Label the labeled data.

[0037] Optionally, for any sensor device j of any type i, determining the abnormal time interval of each sensor device of each type according to the collection time of abnormal data of each sensor device of each type includes:

[0038] Arrange the collection time of all abnormal data of sensor device j in chronological order to obtain the abnormal data collection time series, and initialize the label value of each element in the data collection time series to 0;

[0039] Starting from the first element of the abnormal data acquisition time series, it is used as the processing element; the time difference between the processing element and the next element is determined; if the time difference is not greater than the unit acquisition time difference Δt, then when the processing element is not the penultimate element, the next element is used as the processing element, and the step of determining the time difference between the processing element and the next element and the subsequent steps are repeated. When the processing element is the penultimate element, the processing element and all elements with a label value of 0 before the processing element are grouped together, and the label values of all elements in the group are updated to 1; if the time difference is greater than Δt, then when the processing element is not the penultimate element, the processing element and all elements with a label value of 0 before the processing element are grouped together, and the label values of all elements in the group are updated to 1, and the next element is used as the processing element, and the step of determining the time difference between the processing element and the next element and the subsequent steps are repeated. When the processing element is the penultimate element, the processing element and all elements with a label value of 0 before the processing element are grouped together, and all elements with a label value of 0 after the processing element are grouped together, and the label values of all elements in the two groups are updated to 1;

[0040] Determine the minimum value of each group of elements and maximum value Where x is the group identifier;

[0041] According to each group of elements and Determine the time adjustment coefficient of each group, and determine the time interval corresponding to each group according to the time adjustment coefficient;

[0042] If there are time intervals with non-empty intersections, they are merged to obtain the abnormal time interval of the abnormal sensor device j.

[0043] Optionally, based on the and Determine the time adjustment factor for each group, including:

[0044] according to and The relationship between , and the relationship between the minimum and maximum values of the elements of all groups, determines the time adjustment coefficient of each group.

[0045] Optionally, the labeled data is determined based on the abnormal time interval of each type of sensor device, including:

[0046] Initialize the device marking identifier of each type of sensor device with abnormal data to 0, and initialize the interval marking identifier of the abnormal time interval of each sensor device to 0;

[0047] Select any sensor device with the device mark 0 as the current device;

[0048] Select an abnormal time interval with interval mark 0 in the current device as the current interval;

[0049] Among all non-current devices marked with an identifier of 0, determine whether there is a target interval that has a non-empty intersection with the current interval;

[0050] When the target interval does not exist, the interval marking identifier of the current interval is updated to 1. If the current device has an abnormal time interval with an interval marking identifier of 0, the step of selecting an abnormal time interval with an interval marking identifier of 0 in the current device as the current interval and subsequent steps are re-executed; if the current device does not have an abnormal time interval with an interval marking identifier of 0, the device marking identifier of the current device is updated to 1, and the step of selecting a sensor device with a device marking identifier of 0 as the current device and subsequent steps are re-executed until there is no sensor device with a device marking identifier of 0;

[0051] When a target interval exists, determine whether to mark the current interval based on the target interval and the current interval; update the interval marking identifiers of the target interval and the current interval to 1. If the current device has an abnormal time interval with an interval marking identifier of 0, re-execute the step of selecting an abnormal time interval with an interval marking identifier of 0 in the current device as the current interval and subsequent steps; if the current device does not have an abnormal time interval with an interval marking identifier of 0, update the device marking identifier of the current device to 1, and re-execute the step of selecting a sensor device with a device marking identifier of 0 as the current device and subsequent steps until there is no sensor device with a device marking identifier of 0;

[0052] The data corresponding to the interval determined to be labeled is determined as labeled data.

[0053] Optionally, determining whether to mark the current interval according to the target interval and the current interval includes:

[0054] Determine the intersection interval and union interval of each target interval and the current interval;

[0055] If the ratio of the sum of the time lengths of all intersection intervals to the time length of the current interval is greater than a first ratio threshold, or if there is a target interval in which the ratio of the time length of the intersection interval to the time length of the union interval is greater than a second ratio threshold, it is determined that the current interval is marked.

[0056] (3) Beneficial effects

[0057] The present invention relates to a data processing method for multiple types of sensor equipment based on a field monitoring system, the method comprising: obtaining data collected by each type of sensor equipment; for each type, determining a baseline value for each sensor equipment of that type based on the mean and standard deviation of the data collected by each sensor equipment of that type; determining whether the data collected by each sensor equipment of that type contains abnormal data based on the baseline value; labeling the collected data based on the abnormal data; and performing data processing based on the labeled data. After collecting the data, the method of the present invention does not directly process the data, but rather determines whether there is abnormal data in the collected data, labels the abnormal data, and then performs data processing based on the labeled data. This allows the data to be labeled and focused on the abnormal data during data processing, thereby improving the accuracy of the processing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The present invention provides a flow chart of a data processing method for multiple types of sensor equipment based on a field monitoring system. DETAILED DESCRIPTION

[0059] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation methods in conjunction with the accompanying drawings.

[0060] At present, the data of various types of sensor equipment in the field detection system will be processed directly. In the field monitoring process, the data volume is large and most of the data does not change much. For example, when detecting wild animals, sensor equipment will be installed at the location where the wild animals are found, and the sensor equipment will continuously collect data, and then the wild animals will be found from the data. However, the whereabouts of wild animals are uncertain. Most of the time, wild animals will not appear in the data collection area. At this time, most of the data collected by the sensor equipment is data in the absence of wild animals (this application refers to normal data), and the data with wild animals (this application refers to abnormal data) is very small. If the collected data is processed directly, the data with wild animals will be submerged in the data without wild animals and will be difficult to be found, thereby affecting the accuracy of the processing results.

[0061] Based on this, the present invention relates to a data processing method for multiple types of sensor equipment based on a field monitoring system, the method comprising: obtaining data collected by each type of sensor equipment; for each type, determining a baseline value for each sensor equipment of that type based on the mean and standard deviation of the data collected by each sensor equipment of that type; determining whether the data collected by each sensor equipment of that type contains abnormal data based on the baseline value; labeling the collected data based on the abnormal data; and performing data processing based on the labeled data. After collecting the data, the method of the present invention does not directly process the data, but determines whether there is abnormal data in the collected data, labels the abnormal data, and then performs data processing based on the labeled data. This allows the data to be labeled and focused on the abnormal data during data processing, thereby improving the accuracy of the processing results.

[0062] See also Figure 1 This embodiment provides a data processing method for multiple types of sensor devices based on a field monitoring system, the method comprising:

[0063] 101, obtain data collected by various types of sensor devices.

[0064] There are multiple types of sensor devices, and each type can have multiple sensor devices. The type and number of sensor devices can be determined based on the data collection task. For example, for field environmental monitoring, data can be collected through two temperature sensors and three humidity sensors.

[0065] After collecting data, the sensor devices transmit the data to the data storage device. In step 101 , the data collected by various types of sensor devices can be read from the storage device.

[0066] It should be noted that the device that executes the data processing method for multiple types of sensor devices based on a field monitoring system provided in this embodiment can be the same device as the data storage device, or can be a different device from the data storage device.

[0067] 102. For each type, determine a baseline value for each sensor device of the type based on the mean and standard deviation of the data collected by each sensor device of the type. Determine whether the data collected by each sensor device of the type contains abnormal data based on the baseline value.

[0068] For any type i, the process of determining the baseline value of each sensor device of this type based on the mean and standard deviation of the data collected by each sensor device of this type is:

[0069] For example, for any sensor device j of any type i, the reference value is determined through steps 201 to 208 as follows:

[0070] 201, determine the mean of the data collected by all sensor devices of type i Standard deviation majority

[0071] For example, type i has two sensor devices, namely sensor device j and sensor device j+1. In step 201, the mean of all data collected by sensor device j and sensor device j+1 will be calculated. Standard deviation majority

[0072] 202, determine the mean of the data collected by sensor device j Standard deviation majority

[0073] In step 202, the mean of the data collected by sensor device j is calculated. Standard deviation majority

[0074] 203. Determine the relationship satisfied by the sensing device j.

[0075] when and The relationship between and If the relationship between them is the same, it is determined that the sensing device j satisfies the first type of relationship; otherwise, it is determined that the sensing device j satisfies the second type of relationship.

[0076] In a specific implementation, the first type of relationship consists of two relationships, namely the first relationship and the second relationship. The second type of relationship includes only one relationship, namely the third relationship.

[0077] Characterizes the normal fluctuation range of the data collected by all sensor devices of type i, that is, The data within is normal data. The value that appears most frequently in the data collected by all sensor devices of type i.

[0078] Can be achieved through and relationships, and and The relationship is distinguished between the first and second categories of relationships, and the first and second relationships in the first category of relationships are also distinguished.

[0079] if lie in Neiqie lie in Inside, or and or, and They all think and The relationship between and The relationship between them is the same, then it is determined that the sensing device j satisfies the first type of relationship; otherwise, it is determined that the sensing device j satisfies the second type of relationship.

[0080] For the first type of relationship, if lie in Neiqie lie in If and or, and Then it is determined that the sensing device j satisfies the second relationship. Otherwise, it is determined that the sensing device j satisfies the third relationship.

[0081] If the majority lie in The distribution of all data collected by all sensor devices of type i is normal. lie in This indicates that the distribution of all data collected by sensor device j is normal.

[0082] Therefore, when and When , it means that the data collected by all sensor devices of type i are normal, and the data collected by sensor device j is also normal, then sensor device j satisfies the first relationship, which is that both type i and sensor device j are normal.

[0083] like or This indicates that the distribution of all data collected by all sensor devices of type i is biased to one side. This bias may be affected by extreme values (such as a large number of extremely large data values, or a large number of extremely small data values), which makes the mode Similarly, if or This indicates that the distribution of all data collected by sensor device j is biased towards one side.

[0084] Therefore, when and or, and When , it means that the data collected by all sensor devices of type i are in a skewed distribution, the data collected by sensor device j is in a skewed distribution, and the offset direction of sensor device j is the same as the offset direction of all sensor devices of type i, then sensor device j satisfies the second relationship, which is that both type i and sensor device j are abnormal relationships but the abnormal directions are the same.

[0085] Otherwise, the sensing device j satisfies the third relationship, which is that at least one of the third relationship type i and the sensing device j is abnormal. If both are abnormal, the abnormal directions are different.

[0086] The abnormal relationship represents the relationship between the data distribution of sensor device j and the overall data distribution of type i.

[0087] 204 , sort the data collected by the sensor device j from recent to farthest in time to obtain a collection data sequence of the sensor device j.

[0088] For example, sensor device j collects 4 data, namely data 1 collected at time 1, data 2 collected at time 2, data 3 collected at time 3 and data 4 collected at time 4.

[0089] If time 1 + Δt = time 2, time 2 + Δt = time 3, time 3 + Δt = time 4, where Δt is the unit acquisition time difference Δt, then the acquired data sequence is {data 1, data 2, data 3, data 4}.

[0090] 205, divide the data sequence collected by sensor device j into segments according to the preset time length and determine the mean value of each segment Standard deviation majority

[0091] Among them, u j Segment identifier.

[0092] If the preset time length is Δt*2, the collected data sequence of the sensor device j is divided into two segments, namely segment 1 {data 1, data 2} and segment 2 {data 3, data 4}.

[0093] In step 205 , the mean, standard deviation, and mode of all elements in segment 1 are calculated, and the mean, standard deviation, and mode of all elements in segment 2 are calculated.

[0094] Through step 205 , the data collected by the sensor device j can be subdivided into data collected in multiple collection time periods, and then the mean, standard deviation, and mode of the data collected in each collection time period can be obtained.

[0095] 206, according to the relationship satisfied by the sensor device j and the mean of each segment Standard deviation majority Determine the adjustment coefficient for sensing device j

[0096] If sensor device j satisfies the first type of relationship (such as the first relationship or the second relationship), it means that the distribution of data collected by sensor device j and all sensor devices of type i is the same (either all normal or all abnormal and with the same bias direction), then type i as a whole can be used as the comparison basis for sensor device j. Determine the adjustment coefficient for sensing device j If confirmed

[0097] in, is the ceiling function.

[0098] Characterizes segment u j The degree of dispersion and fluctuation of the data in the field. The smaller the value, the more compact the data and the more stable the fluctuation. In the field monitoring process, the data volume is large and most of the data does not change much. Therefore, the more compact the data and the more stable the fluctuation, the more normal it is. Characterizes the normality of the data collected by sensor device j. Characterizes the overall normality of all sensor devices of type i. If this value is used as the benchmark value of sensor device j, then Characterizes the deviation of sensor device j from the reference value. The larger the value, the more abnormal the sensor device j is. Characterizes the abnormality of sensor device j. The larger the value, the more abnormal it is. The value is an integer, the minimum value is 1, and the maximum value is determined according to the actual situation.

[0099] If sensor device j satisfies the second type of relationship (such as the third relationship), it means that the distribution of data collected by sensor device j and all sensor devices of type i is different (either one is normal and the other is abnormal, or both are abnormal but with different bias directions). In this case, type i as a whole cannot be used as the basis for comparison of sensor device j. Only the data collected by sensor device j can be used to determine Determined based on the standard deviation of the data collected by each sensor device If confirmed

[0100] in, is the minimum value of the standard deviation collected by all sensor devices in class i, is the maximum value of the standard deviation collected by all sensor devices in class i.

[0101] A normalized value that represents the degree of discreteness of the data collected by sensor device j in type i, and is a value between 0 and 1. The normalized value (0-1 value) is mapped to the value of 0.5-1, and we get therefore, This value represents the degree of discreteness of the data collected by sensor device j. The larger the value, the more discrete and abnormal the data collected by sensor device j is. The value is between 0.5 and 1.

[0102] 207, according to the relationship satisfied by the sensor device j and Determine the baseline value of sensing device j

[0103] If the sensor device j satisfies the first type of relationship (such as the first relationship or the second relationship), then according to and each section Determine the baseline value of sensing device j

[0104] For example, according to the relationship satisfied by sensor device j and the mean of each segment Standard deviation majority Determine the calculated value for each segment and the calculated value of sensor device j Determine the baseline value of sensing device j

[0105] Among them, U j is the total number of segments.

[0106] If the sensor device j satisfies the first relationship, it means that the relationship between type i and sensor device j is normal. At this time, the reference value of sensor device j can be calculated by the mean.

[0107] If sensor device j satisfies the second relationship, it means that type i and sensor device j are both abnormal, but the abnormal direction is the same. In this case, the offset needs to be corrected to obtain the reference value.

[0108] For the benchmark value Characterizes the discreteness and fluctuation of the data collected by sensor device j, The larger the value is, the more abnormal the data collected by sensor device j is, and the baseline value needs to be lowered. Characterizes the degree of deviation between the mean value (or adjusted value) of each segment of sensor device j and the overall mean value (or adjusted value), It represents the average deviation of a section, which is used as the adjustment benchmark. Adjust it. When adjusting, The bigger, The smaller, The smaller it is, The bigger, The smaller, that is The smaller it is, the more abnormal the data is and the lower the benchmark value is.

[0109] If the sensor device j satisfies the second type of relationship (such as the third relationship), the reference value of the sensor device j is determined

[0110] Characterizes the discrete degree of the data collected by sensor device j in type i, The larger the value is, the more abnormal the data collected by sensor device j is, and the baseline value needs to be lowered. Characterizes the discrete degree of the sensing device j, which is used as the adjustment benchmark. Adjust it. When adjusting, The bigger, The smaller, The smaller it is, The bigger, The smaller, that is The smaller it is, the more abnormal the data is and the lower the benchmark value is.

[0111] For any type i, the process of determining whether the data collected by each sensor device of that type has abnormal data based on the benchmark value is as follows:

[0112] For any sensor device j in any type i, the data collected by it that does not belong to the normal range is determined as abnormal data.

[0113] Among them, the normal range is based on the reference value of sensor device j Standard deviation of the data collected by sensor device j OK. If the normal range is

[0114] Characterized by the benchmark value The normal distribution interval centered on is normal. The data within this interval is normal data and does not belong to the interval The data is determined to be abnormal data.

[0115] 103. Label the collected data based on the abnormal data.

[0116] When implementing step 103, all abnormal data obtained in step 102 can be marked.

[0117] In a specific implementation, there may be accidental anomalies due to special reasons such as abnormality of a certain sensor device, which may be invalid for data processing. Therefore, step 103 can determine whether the abnormal data in step 102 needs to be marked, and only mark the data that needs to be marked (i.e., data that is not an accidental anomaly). For this situation, the following steps can be implemented:

[0118] 1. Record the collection time of abnormal data of each type of sensor equipment.

[0119] Each sensor device will mark the collection time when collecting data. Here, you can directly obtain the collection time of each abnormal data.

[0120] 2. Determine the abnormal time interval of each type of sensor device based on the collection time of the abnormal data of each type of sensor device.

[0121] For example, for any sensor device j of any type i, the implementation process of this step is as follows:

[0122] 1) Arrange the collection time of all abnormal data of sensor device j in chronological order to obtain the abnormal data collection time series, and initialize the label value of each element in the data collection time series to 0.

[0123] For example, sensor device j has three abnormal data, namely data 1 collected at time 1, data 2 collected at time 2, and data 4 collected at time 4.

[0124] If time 1 + Δt = time 2, time 2 + 2 × Δt = time 4, where Δt is the unit collection time difference Δt, then the abnormal data collection time series is {time 1, time 2, time 4}, and the labeled value of each element in the data collection time series is 0.

[0125] 2) Group the elements of the abnormal data collection time series:

[0126] The grouping process is:

[0127] (1) Start from the first element of the abnormal data collection time series and use it as the processing element.

[0128] (2) Determine the time difference between the processed element and the next element.

[0129] (3) If the time difference is not greater than the unit acquisition time difference Δt, then:

[0130] A. When the processing element is not the penultimate element, the element following the processing element is used as the processing element, and the step of determining the time difference between the processing element and the element following it (ie (2)) and subsequent steps are repeatedly executed.

[0131] B. When the processing element is the second-to-last element, group the processing element and all elements whose label values before the processing element are 0, and update the label values of all elements in the group to 1.

[0132] (4) If the time difference is greater than Δt, then:

[0133] A. When the processing element is not the penultimate element, group the processing element and all elements whose label value is 0 before the processing element, update the label value of all elements in the group to 1, use the element after the processing element as the processing element, and repeat the step of determining the time difference between the processing element and the element after it (i.e., (2)) and subsequent steps.

[0134] B. When the processing element is the second to last element, group the processing element and all elements with a label value of 0 before the processing element into one group, group all elements with a label value of 0 after the processing element into one group, and update the label values of all elements in the two groups to 1.

[0135] For example, the abnormal data collection time series is {time 1, time 2, time 4}, and the first element of the abnormal data collection time series (ie, time 1) is used as the processing element.

[0136] Determine the time difference (i.e., Δt) between the processing element (i.e., time 1) and the element following it (i.e., time 2). This time difference = Δt (i.e., the time difference is not greater than Δt), and the processing element (i.e., time 1) is the second-to-last element in the time series of non-abnormal data collection. Therefore, the element following the processing element (i.e., time 2) is used as the processing element.

[0137] Determine the time difference between the processing element (i.e., time 2) and the element following it (i.e., time 3) (i.e., 2 × Δt). This time difference is greater than Δt, and the processing element (i.e., time 2) is the second-to-last element in the time series of the abnormal data collection. Therefore, the processing element (i.e., time 2) is grouped with all elements labeled 0 before the processing element (i.e., time 1), and all elements labeled 0 after the processing element (i.e., time 4) are grouped together. This forms two groups: the first group includes time 1 and time 2, and the second group includes time 4. Update the labeled values of all elements in both groups (i.e., time 1, time 2, and time 4) to 1.

[0138] The above grouping process can group abnormal data collected at similar times into one group, so that each group can represent all data involved in an abnormality.

[0139] 3) Determine the minimum value of each group of elements and maximum value

[0140] Where x is the group identifier.

[0141] If there is only one element in a group (like the second group), then (i.e. time 4).

[0142] If there are multiple elements in a group (such as the first group), the smallest value of the elements in the group is determined as (like ), the minimum value of the element is determined as (like ).

[0143] 4) According to each group of elements and Determine the time adjustment coefficient for each group, and determine the time interval corresponding to each group based on the time adjustment coefficient.

[0144] Among them, according to and The relationship between the minimum and maximum values of the elements of all groups is used to determine the time adjustment coefficient of each group. For example, if the time adjustment coefficient of the xth group of sensor device j is but

[0145] in, For each group The minimum value of For each group The maximum value of , X is the number of groups.

[0146] The time interval corresponding to each group can be determined as

[0147] Among them, t b is the time of the first data collected by sensor device j, t w is the time of the last data collected by sensor device j, max{·} is the maximum value function, and min{·} is the minimum value function.

[0148] The sensor device j collects 4 data, namely data 1, data 2, data 3, and data 4, and the time of the first data (data 1) collected by the sensor device j is time 1 (t b = time 1), the time of the last data (i.e. data 4) collected by sensor device j is time 4 (i.e. t w = time 4). The abnormal data collection time series is {time 1, time 2, time 4}, forming two groups, the first group includes time 1 and time 2, and the second group includes time 4. For example, X = 2, X-1 = 1,

[0149] For the first group, x=1, The time interval corresponding to the first group is

[0150] For the second group, x=2, The time interval corresponding to the first group is

[0151] Each group represents a data anomaly, which can be a regular anomaly caused by normal reasons or an accidental anomaly due to uncontrollable factors. To accurately determine the cause of the anomaly during data processing, the time interval corresponding to each group element needs to be reasonably expanded to include some normal data. The process of changing from normal data to abnormal data can then be used to assess whether the anomaly is accidental.

[0152] Adjust coefficients when expanding It is realized based on Δt. The main considerations when adjusting The value of . It represents the ratio of the duration of the data set x to the duration of all abnormal data of the sensor device h. The smaller the ratio, the shorter the duration of the abnormality (it may be an accidental abnormality). More data is needed to determine whether the abnormality is caused by accidental reasons. Therefore, the adjustment coefficient It will get bigger, thus increasing the time interval.

[0153] 5) If there is a non-empty time interval with an intersection, it means that the collection time of the two abnormal data overlaps. The two groups may be caused by the same abnormality, so they will be merged to obtain the abnormal time interval of the abnormal sensor device j.

[0154] At this point, we can obtain the time involved in each abnormal data, that is, the abnormal time interval.

[0155] 3. Determine the labeled data based on the abnormal time interval of each type of sensor equipment.

[0156] The implementation process of this step is as follows:

[0157] 1) Initialize the device labeling identifier of each type of sensor device with abnormal data to 0, and initialize the interval labeling identifier of the abnormal time interval of each sensor device to 0.

[0158] The device marking identifier is used to identify whether the corresponding sensing device has completed the marking determination, and the interval marking identifier is used to identify whether the corresponding interval has completed the marking determination.

[0159] For example, there are three sensor devices with abnormal data: sensor 1, sensor 2, and sensor 3. Sensor 1 and sensor 2 are of type 1, while sensor 3 is of type 2. Furthermore, sensor 1 has three abnormal time intervals: interval 11, interval 12, and interval 13; sensor 2 has two abnormal time intervals: interval 21 and interval 22; and sensor 3 has one abnormal time interval: interval 31.

[0160] In this step, the device labeling identifiers of sensor device 1, sensor device 2 and sensor device 3 are all initialized to 0, and the interval labeling identifiers of interval 11, interval 12, interval 13, interval 21, interval 22 and interval 31 are initialized to 0.

[0161] 2) Select any sensor device with the device identifier 0 as the current device.

[0162] For example, sensor device 1 is selected as the current device.

[0163] 3) Select an abnormal time interval with interval mark 0 in the current device as the current interval.

[0164] For example, select interval 11 as the current interval.

[0165] 4) Among all non-current devices marked with an identifier of 0, determine whether there is a target interval whose intersection with the current interval is not empty.

[0166] For example, in intervals 21, 22, and 31, determine the intersection of interval 21 and interval 11 (recorded as intersection 1), the intersection of interval 22 and interval 11 (recorded as intersection 2), and the intersection of interval 31 and interval 11 (recorded as intersection 3).

[0167] If intersection 1 is empty, intersection 2 is not empty, and intersection 3 is not empty, the target intervals are interval 22 and interval 31.

[0168] 5) When the target interval does not exist, it means that only the current device has abnormal data in this time interval. When conducting field monitoring, various types of sensor devices are monitored comprehensively from multiple angles at the same time. When an abnormal event occurs, it is difficult for only one sensor device to have abnormal data. Generally, multiple sensor devices have abnormal data in this time period. If there is no target interval, it can be considered that the current sensor device may have an unexpected abnormality caused by uncontrollable factors in this time period. Therefore, it is considered that the current interval has completed the determination of whether it needs to be marked, and then the following steps are performed:

[0169] (1) Update the interval label of the current interval to 1.

[0170] (2) If the current device has an abnormal time interval with an interval marking of 0, it means that the current device still has an unprocessed abnormal time interval (that is, an abnormal time interval that has not been confirmed to be marked). At this time, the step of selecting an abnormal time interval with an interval marking of 0 in the current device as the current interval (that is, 3)) and subsequent steps will be re-executed to process the unprocessed interval (that is, determine whether it needs to be marked).

[0171] If the current device does not have an abnormal time interval with an interval marking identifier of 0, it means that all abnormal time intervals of the current device have been processed (that is, the determination of whether labeling is required has been completed). At this time, the device labeling identifier of the current device is updated to 1 (indicating that the current device has completed the determination of whether labeling is required), and the step of selecting any sensor device with a device marking identifier of 0 as the current device (that is, 2) and subsequent steps are re-executed, and other sensor devices are processed (that is, whether labeling is required is determined) until there is no sensor device with a device marking identifier of 0.

[0172] 6) If the target interval exists, it means that more than one sensor device has abnormal data in this time period. In this case, perform the following steps:

[0173] (1) Determine whether to mark the current interval based on the target interval and the current interval.

[0174] In a specific implementation, the intersection interval and union interval of each target interval and the current interval can be determined. If the ratio of the sum of the durations of all the intersection intervals to the duration of the current interval is greater than a first ratio threshold, or if there is a target interval where the ratio of the duration of the intersection interval to the duration of the union interval is greater than a second ratio threshold, then the current interval is determined to be marked.

[0175] The intersection interval of any target interval and the current interval indicates the length of their overlap. A longer interval indicates more time spent experiencing anomalies simultaneously. The union interval of any target interval and the current interval indicates the total duration of the data anomaly. For the current interval, if there are many target intervals, this indicates that more sensor devices are experiencing data anomalies during the same time period. Therefore, this time period is an anomaly and needs to be labeled. If there are few target intervals, but the time spent experiencing anomalies simultaneously with the target interval and the current interval (i.e., the longer the intersection interval), this indicates that the data collected by both intervals is anomaly simultaneously. This is not a random occurrence and requires labeling. Both the large number of target intervals and the length of time during which anomalies occur are reflected in the sum of the lengths of all the intersection intervals. A large number of target intervals increases the sum of the lengths of all the intersection intervals, and a longer period of simultaneous anomalies also increases the sum of the lengths of all the intersection intervals. Therefore, if the sum of the lengths of all the intersection intervals is high, it is confirmed that the time period is an anomaly and needs to be labeled. For example, according to the accuracy requirements of the monitoring task, a first ratio threshold is set (the threshold is a number between 0 and 1, such as 0.7. The larger the first ratio threshold, the fewer time periods are marked, and more intersection time is required for marking. Some intervals will be missed, which will reduce the accuracy of the final data processing. However, if there is less marked data, then there is less data to pay attention to in subsequent data processing, which will increase the speed of data processing. Therefore, the first ratio threshold can be comprehensively determined based on the efficiency and accuracy requirements of the specific monitoring task). If the ratio of the sum of the time lengths of all intersection intervals to the time length of the current interval is greater than the first ratio threshold, it means that the time period is an abnormal time period and needs to be marked.

[0176] The ratio of the sum of the time lengths of all the intersection intervals to the time length of the current interval is greater than the first ratio threshold, which determines whether the current interval needs to be marked from the perspective of all target intervals as a whole.

[0177] It is also possible to determine whether the current interval needs to be labeled from the perspective of a single target interval. That is, if there is a target interval, and the ratio of the time length of the intersection interval of the intersection interval and the current interval to the time length of the union interval of the intersection interval and the current interval is relatively large, it means that the longer the target interval and the current interval are abnormal at the same time, the more abnormal the data collected by the two intervals are synchronously, and it is not a random anomaly, so it needs to be labeled. For example, based on the accuracy requirements of the monitoring task, a second ratio threshold is set (this threshold is a number between 0 and 1, such as 0.7. The larger the second ratio threshold, the fewer time periods are labeled, and more intersection time periods are required for labeling. Some intervals will be missed, which will reduce the accuracy of the final data processing. However, less labeled data means less data needs to be focused on in subsequent data processing, which will increase the speed of data processing. Therefore, the second ratio threshold can be comprehensively determined based on the efficiency and accuracy requirements of the specific monitoring task). If there is a target interval, if the ratio of the time length of its intersection interval to the time length of the union interval is greater than the second ratio threshold, it means that the time period is an abnormal time period and needs to be labeled.

[0178] It should be noted that this embodiment does not limit the relationship between the first ratio threshold and the second ratio threshold. The first ratio threshold may be greater than the second ratio threshold, the first ratio threshold may be less than the second ratio threshold, or the first ratio threshold may be equal to the second ratio threshold.

[0179] (2) Update the interval marking identifiers of the target interval and the current interval to 1.

[0180] Therefore, at this time, both the target interval and the current interval have been processed (ie, the determination of whether they need to be marked has been completed), so the interval marking flags of the target interval and the current interval are both updated to 1.

[0181] (3) If the current device has an abnormal time interval with an interval marking of 0, it means that the current device still has an unprocessed abnormal time interval (that is, an abnormal time interval that has not been confirmed to be marked). At this time, the step of selecting an abnormal time interval with an interval marking of 0 in the current device as the current interval (that is, 3)) and subsequent steps will be re-executed to process the unprocessed interval (that is, determine whether it needs to be marked).

[0182] If the current device does not have an abnormal time interval with an interval marking of 0, it means that all abnormal time intervals of the current device have been processed (that is, the determination of whether labeling is required has been completed). At this time, the device labeling identifier of the current device is updated to 1 (indicating that the current device has completed the determination of whether labeling is required), and the step of selecting any sensor device with a device marking of 0 as the current device (that is, 2) and subsequent steps are re-executed until there is no sensor device with a device marking of 0.

[0183] 7) The data corresponding to the marked interval is determined as the marked data.

[0184] 4. Label the labeled data.

[0185] At this point, the abnormal data can be filtered, and the time intervals where unintentional abnormalities occurred can be identified as marked intervals. The data corresponding to these marked intervals can then be labeled. This ensures that the labeled data is accurate and not accidental, thus ensuring the accuracy of subsequent data processing results. Furthermore, by labeling all non-abnormal data, non-accidental abnormal data will not be labeled, thus reducing the amount of data labeled. This also reduces the amount of data that needs to be processed during subsequent data processing, improving data processing efficiency.

[0186] 104 , performing data processing based on the labeled data.

[0187] After obtaining abnormal data, the abnormal data can be marked. After marking the abnormal data, data processing is performed based on the marked data.

[0188] The labeled abnormal data is data that is different from the majority of the data. This data is likely to be the data that the inspection task is concerned about. Therefore, when processing the data, you can consider focusing on the analysis based on the data labeling to avoid the abnormal data being submerged in the normal data and being ignored during data processing, thereby improving the accuracy of the processing results.

[0189] In addition, the data processing scheme adopts the existing scheme. For example, if the data collection task is to conduct field environmental monitoring, then the data processing is to calculate the environmental monitoring results. The data processing scheme is not limited here and will not be explained in detail.

[0190] This embodiment provides a data processing method for multiple types of sensor equipment based on a field monitoring system, which obtains data collected by each type of sensor equipment; for each type, determines the baseline value of each sensor equipment of that type based on the mean and standard deviation of the data collected by each sensor equipment of that type; determines whether the data collected by each sensor equipment of that type contains abnormal data based on the baseline value; labels the collected data based on the abnormal data; and performs data processing based on the labeled data. After collecting the data, the method of this embodiment does not directly process the data, but determines whether there is abnormal data in the collected data, labels the abnormal data, and then processes the data based on the labeled data. This allows the data to be processed based on the labeling of the data, focusing on the abnormal data, thereby improving the accuracy of the processing results.

[0191] The technical principles of the present invention have been described in conjunction with specific embodiments. These descriptions are intended solely to illustrate the principles of the present invention and are not to be construed in any way as limiting the scope of protection of the present invention. Based on the explanations herein, those skilled in the art will be able to devise other specific embodiments of the present invention without inventive effort, and such embodiments will fall within the scope of protection of the present invention.

Claims

1. A data processing method for multiple types of sensor equipment based on a field monitoring system, characterized in that: The method comprises: Obtain data collected by various types of sensor devices; For each type, a baseline value of each sensor device of the type is determined based on the mean and standard deviation of the data collected by each sensor device of the type; and whether the data collected by each sensor device of the type contains abnormal data is determined based on the baseline value; Annotate the collected data based on abnormal data; Perform data processing based on the labeled data.

2. The method according to claim 1, characterized in that For any type i, determining the baseline value of each sensor device of this type based on the mean and standard deviation of the data collected by each sensor device of this type includes: For any sensor device j of any type i, the baseline value is determined by the following steps: Determine the mean of the data collected by all sensor devices of type i Standard deviation majority Determine the mean of the data collected by sensor device j Standard deviation majority Determine the relationship that the sensing device j satisfies; where and The relationship between and If the relationship between them is the same, it is determined that the sensor device j satisfies the first type of relationship; otherwise, it is determined that the sensor device j satisfies the second type of relationship; Sort the data collected by sensor device j from recent to farthest in time to obtain the collected data sequence of sensor device j; The data sequence collected by the sensor device j is segmented according to the preset time length, and the mean value of each segment is determined. Standard deviation majority Among them, u j is the segment identifier; According to the relationship satisfied by the sensing device j and the Determine the adjustment coefficient for sensing device j According to the relationship satisfied by the sensor device j and Determine the baseline value of sensing device j 3. The method according to claim 2, characterized in that The relationship between the sensing device j and each segment Determine the adjustment coefficient for sensing device j include: If the sensor device j satisfies the first type of relationship, then according to the Determine the adjustment coefficient for sensing device j If the sensor device j satisfies the second type of relationship, the standard deviation of the data collected by each sensor device is used to determine 4. The method according to claim 2, characterized in that The relationship satisfied by the sensing device j and Determine the baseline value of sensing device j include: If the sensor device j satisfies the first type of relationship, then according to and each section Determine the baseline value of sensing device j If the sensing device h satisfies the second type of relationship, then according to and Determine the baseline value of sensing device j 5. The method according to claim 1, wherein For any type i, determining whether the data collected by each sensor device of this type contains abnormal data based on the reference value includes: For any sensor device h of any type i, the data collected by it that does not fall within the normal range is determined as abnormal data; The normal interval is determined based on the reference value of the sensor device j and the standard deviation of the data collected by the sensor device j.

6. The method according to claim 1, characterized in that The labeling of the collected data based on the abnormal data includes: Record the collection time of abnormal data of each type of sensor equipment; Determine the abnormal time interval of each type of sensor device according to the collection time of abnormal data of each type of sensor device; Determine the labeled data based on the abnormal time interval of each type of sensor equipment; Label the labeled data.

7. The method according to claim 6, characterized in that For any sensor device j of any type i, determining the abnormal time interval of each sensor device of each type according to the collection time of abnormal data of each sensor device of each type includes: Arrange the collection time of all abnormal data of sensor device j in chronological order to obtain the abnormal data collection time series, and initialize the label value of each element in the data collection time series to 0; Starting from the first element of the abnormal data acquisition time series, it is used as the processing element; the time difference between the processing element and the next element is determined; if the time difference is not greater than the unit acquisition time difference Δt, then when the processing element is not the penultimate element, the next element is used as the processing element, and the step of determining the time difference between the processing element and the next element and the subsequent steps are repeated. When the processing element is the penultimate element, the processing element and all elements with a label value of 0 before the processing element are grouped together, and the label values of all elements in the group are updated to 1; if the time difference is greater than Δt, then when the processing element is not the penultimate element, the processing element and all elements with a label value of 0 before the processing element are grouped together, and the label values of all elements in the group are updated to 1, and the next element is used as the processing element, and the step of determining the time difference between the processing element and the next element and the subsequent steps are repeated. When the processing element is the penultimate element, the processing element and all elements with a label value of 0 before the processing element are grouped together, and all elements with a label value of 0 after the processing element are grouped together, and the label values of all elements in the two groups are updated to 1; Determine the minimum value of each group of elements and maximum value Where x is the group identifier; According to each group of elements and Determine the time adjustment coefficient of each group, and determine the time interval corresponding to each group according to the time adjustment coefficient; If there are time intervals with non-empty intersections, they are merged to obtain the abnormal time interval of the abnormal sensor device j.

8. The method according to claim 7, characterized in that According to each group of elements and Determine the time adjustment factor for each group, including: according to and The relationship between , and the relationship between the minimum and maximum values of the elements of all groups, determines the time adjustment coefficient of each group.

9. The method according to claim 6, characterized in that The step of determining the labeled data based on the abnormal time interval of each type of sensor device includes: Initialize the device marking identifier of each type of sensor device with abnormal data to 0, and initialize the interval marking identifier of the abnormal time interval of each sensor device to 0; Select any sensor device with the device mark 0 as the current device; Select an abnormal time interval with interval mark 0 in the current device as the current interval; Among all non-current devices marked with an identifier of 0, determine whether there is a target interval that has a non-empty intersection with the current interval; When the target interval does not exist, the interval marking identifier of the current interval is updated to 1. If the current device has an abnormal time interval with an interval marking identifier of 0, the step of selecting an abnormal time interval with an interval marking identifier of 0 in the current device as the current interval and subsequent steps are re-executed; if the current device does not have an abnormal time interval with an interval marking identifier of 0, the device marking identifier of the current device is updated to 1, and the step of selecting a sensor device with a device marking identifier of 0 as the current device and subsequent steps are re-executed until there is no sensor device with a device marking identifier of 0; When a target interval exists, determine whether to mark the current interval based on the target interval and the current interval; update the interval marking identifiers of the target interval and the current interval to 1. If the current device has an abnormal time interval with an interval marking identifier of 0, re-execute the step of selecting an abnormal time interval with an interval marking identifier of 0 in the current device as the current interval and subsequent steps; if the current device does not have an abnormal time interval with an interval marking identifier of 0, update the device marking identifier of the current device to 1, and re-execute the step of selecting a sensor device with a device marking identifier of 0 as the current device and subsequent steps until there is no sensor device with a device marking identifier of 0; The data corresponding to the interval determined to be labeled is determined as labeled data.

10. The method according to claim 9, characterized in that The determining whether to mark the current interval according to the target interval and the current interval includes: Determine the intersection interval and union interval of each target interval and the current interval; If the ratio of the sum of the time lengths of all intersection intervals to the time length of the current interval is greater than a first ratio threshold, or if there is a target interval in which the ratio of the time length of the intersection interval to the time length of the union interval is greater than a second ratio threshold, it is determined that the current interval is marked.

Images