Sensor data compression storage method and system

Abstract

The application relates to a sensor data compression storage method and system, which comprises the following steps: collecting sensor data based on a set sampling interval, receiving current sampling data; calculating the difference between the current data and the last valid sampling data to obtain the current data difference; calling the written difference sequence, calculating the change rate of the current data difference; judging whether the current data difference is greater than the dynamic difference threshold and whether the difference change rate is greater than the change rate threshold; if any condition is met, the current sampling data is regarded as valid sampling data, and the current sampling data, the difference and the sampling interval are written into the cache; otherwise, the current sampling data is discarded; judging whether the last valid sampling data is the first sampling data of the day, if not, the last valid sampling data is erased, and if yes, the last valid sampling data is reserved; when a preset storage condition is triggered, the cache area data is written into the storage area to form compressed data; receiving an uploading instruction from an upper computer, uploading the compressed data, and reconstructing the compressed data in the upper computer through linear interpolation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a sensor data compression storage method and system. BACKGROUND

[0002] With the rapid development of Internet of Things (IoT) technology, various environmental sensors such as temperature, humidity, and pressure are widely used in smart agriculture, remote monitoring, industrial automation, and other scenarios. Since terminal devices are often deployed in remote or complex environments, the data collected by sensors usually needs to be stored in the local storage medium (such as Flash or EEPROM) of a microcontroller (MCU) for a long time to ensure data integrity and traceability in poor communication or offline scenarios.

[0003] However, the existing data storage method has many shortcomings:

[0004] The original data collected by the sensor is usually stored in the form of complete timestamps and floating-point values, and each record occupies about 6 to 12 bytes of space, which severely limits the storage period under the limited storage capacity of embedded devices. Although existing general compression algorithms (such as ZIP, LZ4, etc.) can reduce storage, their computational complexity is high, and they require a large amount of RAM and CPU resources, which is not suitable for low-end MCU environments such as Cortex-M0 / M3. At the same time, even slight fluctuations in sensor data are frequently written to Flash, causing write amplification and accelerating the aging of the storage medium; and the Flash itself has a limited erase-write lifespan (usually about 100,000 times), which further shortens the service life of the device. At the same time, traditional data collection methods often use fixed thresholds or fixed sampling intervals, lacking adaptive adjustment to dynamic changes in the environment, resulting in low storage and energy efficiency.

[0005] How to efficiently and reliably store and manage sensor data locally under the conditions of low power consumption, low computing power, and limited storage has become a technical problem that needs to be solved in the field of Internet of Things devices. SUMMARY

[0006] To at least partially overcome the problem of being difficult to efficiently and reliably store and manage sensor data locally under the conditions of low power consumption, low computing power, and limited storage in the related art, the present application provides a sensor data compression storage method and system.

[0007] The scheme of the present application is as follows:

[0008] According to a first aspect of an embodiment of the present application, a sensor data compression storage method is provided, comprising:

[0009] controlling the sensor to sample data based on a set sampling interval, receiving current sampling data input by the sensor;

[0010] calculating a data difference value between the current sampling data and previous valid sampling data, denoted as a current data difference value;

[0011] calling the written data difference value sequence, and calculating a difference change rate of the current data difference value;

[0012] judging whether the current data difference value is greater than a dynamic difference value threshold and whether the difference change rate is greater than a change rate threshold; wherein the dynamic difference value threshold is dynamically adjusted every interval of a preset number of data samplings;

[0013] if the current data difference value is greater than the dynamic difference value threshold and / or the difference change rate is greater than the change rate threshold, regarding the current sampling data as valid sampling data, and writing the current sampling data, the current data difference value and the current sampling interval into a cache area; otherwise, regarding the current sampling data as invalid sampling data and discarding the current sampling data;

[0014] judging whether the previous valid sampling data is the first sampling data of a current day;

[0015] if the previous valid sampling data is not the first sampling data of the current day, erasing the previous valid sampling data; if the previous valid sampling data is the first sampling data of the current day, not erasing the previous valid sampling data;

[0016] when a preset storage condition is triggered, writing cache data in the cache area into a storage area to form compressed data;

[0017] receiving an upload instruction sent by an upper computer, and uploading the compressed data in the storage area to the upper computer; the compressed data is reconstructed in the upper computer by a linear interpolation method.

[0018] Preferably, the method further comprises:

[0019] if the current sampling data is the first sampling data of the current day, regarding the current sampling data as valid sampling data directly, and additionally writing a time stamp of the current sampling data.

[0020] Preferably, the dynamic difference value threshold is dynamically adjusted every interval of a preset number of data samplings, comprising:

[0021] taking a preset number of data samplings as a threshold adjustment period, calling a data difference value sequence in a current threshold adjustment period, and constructing a data difference absolute value sequence;

[0022] calculating a mean absolute deviation of the data difference absolute value sequence;

[0023] calculating a dynamic difference value threshold of a next threshold adjustment period according to the mean absolute deviation, a basic difference value threshold and an adjustment coefficient:

[0024] T = B x (1 + a x M);

[0025] wherein, T represents the dynamic difference threshold value of the next threshold adjustment period; B represents the basic difference threshold value; a represents the adjustment coefficient, and the value is 0.5-1.0; M represents the average absolute deviation;

[0026] determining whether the dynamic difference threshold value of the next threshold adjustment period is in the set dynamic difference threshold value interval;

[0027] if the dynamic difference threshold value of the next threshold adjustment period is greater than the set maximum dynamic difference threshold value, the dynamic difference threshold value of the next threshold adjustment period is adjusted to the set maximum dynamic difference threshold value;

[0028] if the dynamic difference threshold value of the next threshold adjustment period is less than the set minimum dynamic difference threshold value, the dynamic difference threshold value of the next threshold adjustment period is adjusted to the set minimum dynamic difference threshold value.

[0029] Preferably, the method further comprises:

[0030] entering the current day, setting the daytime basic difference threshold value and the nighttime basic difference threshold value of the current day according to the current seasonal information and the current location information.

[0031] Preferably, the method further comprises:

[0032] controlling the sensor to sample data based on the initial sampling interval;

[0033] record the number of invalid data with continuity by the counter;

[0034] when the value recorded in the counter reaches the low-power consumption trigger threshold, multiply the initial sampling interval by the interval coefficient, and control the sensor to sample data based on the updated sampling interval; the interval coefficient is greater than 1.

[0035] Preferably, the method further comprises:

[0036] if the updated sampling interval exceeds the preset maximum sampling interval, cancel the update of the sampling interval.

[0037] Preferably, the method further comprises:

[0038] if the current sampling data is written into the cache area, clear the record of the counter.

[0039] Preferably, the preset storage condition at least includes:

[0040] the current data difference is greater than the product of the dynamic difference threshold and the storage coefficient, and / or the difference change rate is greater than the product of the change rate threshold and the storage coefficient;

[0041] a set storage time is reached;

[0042] a buffer usage rate exceeds a preset cache threshold;

[0043] the system is about to enter a deep sleep mode.

[0044] Preferably, the method further comprises:

[0045] the total number of collected data, the number of valid data, and the number of storage area write times are counted as local statistical indicators;

[0046] the local statistical indicators are sent to the upper computer, so that the upper computer calculates the valid data proportion, the number and proportion of invalid data, the average number of data per write in the storage area, the original storage estimate, the compressed storage estimate, and the storage space saving ratio according to the local statistical indicators.

[0047] According to a second aspect of the embodiments of the present application, a sensor data compression storage system is provided, comprising:

[0048] a local processor and an upper computer;

[0049] the local processor and the upper computer are connected through wireless communication:

[0050] the local processor is used to execute a sensor data compression storage method as described in any one of the above.

[0051] The technical solutions provided by the present application can include the following beneficial effects:

[0052] The difference between the sensor sampling data and the last valid write data and the difference change rate are jointly determined, and only when the dynamic difference threshold or the change rate threshold is exceeded, the cache is written, which effectively avoids the space waste caused by storing each record of the original data. Compared with directly storing the complete timestamp and floating point value of each record, the present technical solution can significantly reduce the storage data volume, greatly improve the utilization rate of limited storage resources, and thus prolong the local storage period.

[0053] In the present technical solution, when the current sampling data is valid sampling data, the current sampling data, the current data difference, and the current sampling interval are written into the cache area, and the last valid sampling data is erased. The final storage content only includes the first sampling data of the current day and the difference and sampling interval corresponding to the subsequent valid sampling data, which can further save storage space.

[0054] The effective data is temporarily stored in the cache area, and is written into the storage area when the preset storage condition is triggered, so that batch writing operation is realized. This method avoids the write amplification effect caused by frequent writing triggered by slight changes, reduces the erase-write frequency of the storage area, and significantly prolongs the service life of the memory device.

[0055] The dynamic difference threshold changing over time is introduced, which can be adaptively adjusted in combination with different environmental states. When the environmental data fluctuates greatly, the threshold is raised to avoid invalid writing; when the environment is stable, the threshold is lowered to improve the sensitivity to slight effective changes, so as to balance between compression efficiency and data fidelity.

[0056] The compressed data is reconstructed by the upper computer using linear interpolation method, which can greatly reduce the storage and writing times while ensuring the continuity and trend integrity of the data sequence, and takes into account the storage efficiency and data availability. At the same time, the local device is only responsible for reporting the compressed original data packet, thereby maximizing the saving of local computing resources and wireless communication bandwidth. This architecture also makes the future upgrade of the reconstruction algorithm unnecessary to change the local firmware, and has high maintainability.

[0057] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0058] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0059] Figure 1 is a flow diagram of a sensor data compression storage method provided by an embodiment of the present application;

[0060] Figure 2 is a comparison diagram of effective data, invalid data and upper computer reconstruction interpolation data provided by an embodiment of the present application;

[0061] Figure 3 is a structural diagram of a sensor data compression storage system provided by an embodiment of the present application.

[0062] Reference signs: DETAILED DESCRIPTION

[0063] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The description below refers to the accompanying drawings, which show, by way of example, specific embodiments with which this application can be practiced. The following detailed description is not intended to limit the scope of the application, as claimed, but is merely representative of some embodiments consistent with the principles of the application. While the following exemplary embodiments are described in the context of a device and method for use with a wireless communication system, various changes and modifications could be suggested to one skilled in the art. Additionally, various other embodiments are within the scope of the application. It is therefore intended that the application not be limited to the embodiments disclosed for illustrative purposes.

[0064] Figure 1 is a flowchart of a sensor data compression storage method provided by an embodiment of the application, referring to Figure 1 A sensor data compression storage method comprises:

[0065] S11: controlling the sensor to sample data based on a set sampling interval, receiving current sampling data input by the sensor;

[0066] The sensor periodically samples environmental parameters such as temperature, humidity, and pressure.

[0067] The local processor receives current sampling data input by the sensor.

[0068] The local processor can be an MCU or an embedded chip.

[0069] S12: calculating a data difference value between the current sampling data and the last valid sampling data, denoted as a current data difference value;

[0070] The difference value is used to reflect the change amplitude of the current sensor data relative to the last valid written data.

[0071] Preferably, the data difference value is converted into an 8-bit signed integer representation (e.g. 0.1°C / LSB) in proportion.

[0072] S13: retrieving a sequence of written data difference values, calculating a difference value change rate of the current data difference value;

[0073] The difference value change rate can reflect whether the current data change has a tendency to intensify or slow down, and is used in the present technical solution to capture sudden changes in data trends, such as the inflection point from slow rise to rapid rise, enhancing the sensitivity of the algorithm to sudden events.

[0074] S14: determining whether the current data difference value is greater than a dynamic difference value threshold and whether the difference value change rate is greater than a change rate threshold; wherein the dynamic difference value threshold is dynamically adjusted after every preset number of data sampling intervals;

[0075] S15: If the current data difference is greater than the dynamic difference threshold value, and / or, the difference rate is greater than the rate threshold value, the current sampling data is regarded as valid sampling data, and the current sampling data, the current data difference and the current sampling interval are written into the cache area; otherwise, the current sampling data is regarded as invalid sampling data and discarded;

[0076] The cache area is generally implemented by RAM, and is used to temporarily save candidate data. The size of the cache area is configured according to actual local resources.

[0077] When the local processor judges that the current sampling data meets the valid condition and writes into the cache area, the following three types of information are also stored:

[0078] The current sampling data itself, that is, the physical quantity value actually measured by the sensor (such as temperature, humidity or pressure value);

[0079] The current data difference, that is, the difference between the current sampling data and the last valid written data, which can reflect the data change amplitude;

[0080] The current sampling interval, that is, the time interval between the current sampling and the last valid sampling, which is used to represent the sparsity in the time dimension.

[0081] This design helps the host computer to use the data difference and the sampling interval to perform more accurate sequence reconstruction. For example, when the original sampling frequency is high but the valid points are sparse, the stored sampling interval can help the host computer to accurately restore the time axis distribution; and the stored data difference can provide a direct basis for subsequent statistical analysis and dynamic threshold adjustment.

[0082] S16: Judge whether the last valid sampling data is the first sampling data of the current day;

[0083] S17: If the last valid sampling data is not the first sampling data of the current day, the last valid sampling data is erased; if the last valid sampling data is the first sampling data of the current day, no erasing is performed;

[0084] S18: When the preset storage condition is triggered, the cache data of the cache area is written into the storage area to form compressed data;

[0085] In specific practice, in addition to the first sampling data of the current day, each valid data written into the storage area contains 1 byte of data difference and 4-bit time interval code (such as 1s, 5s, 10s, 1min, 5min, 10min, 30min, 1h, etc. Common periods), which occupies 1.5 bytes in total.

[0086] And the first sampling data of the current day needs to write the complete data value and the time stamp.

[0087] S19: receiving the uploading instruction sent by the host computer, uploading the compressed data in the storage area to the host computer; the compressed data is reconstructed in the host computer by linear interpolation method.

[0088] The compressed data is reconstructed in the host computer by linear interpolation method to restore the complete sequence close to the original sensor data, thereby ensuring data availability.

[0089] Embodiment Two

[0090] It should be noted that the method further comprises:

[0091] If the current sampling data is the first sampling data of the current day, it is directly regarded as valid sampling data, and the timestamp of the current sampling data is additionally written.

[0092] Directly regarding the first sampling data of the current day as valid sampling data for writing can simplify the processing logic. Writing the timestamp of the first sampling data of the current day can determine the time reference point.

[0093] In the technical solution, when the current sampling data is valid sampling data, the current sampling data, the current data difference value and the current sampling interval are written into the cache area, and the last valid sampling data is erased (if the last valid sampling data is the first sampling data of the current day, it is not erased). The final storage content only includes the first sampling data of the current day and the difference value and sampling interval corresponding to the subsequent valid sampling data, so that the storage space can be further saved.

[0094] The host computer can reconstruct the subsequent valid sampling data according to the difference value corresponding to the first sampling data of the current day and the subsequent valid sampling data.

[0095] Embodiment Three

[0096] It should be noted that the method further comprises:

[0097] Taking the preset data sampling number as a threshold adjustment period, calling the data difference value sequence in the current threshold adjustment period, and constructing a data difference value absolute value sequence;

[0098] Calculating the average absolute deviation of the data difference value absolute value sequence;

[0099] According to the average absolute deviation, the basic difference value threshold and the adjustment coefficient, the dynamic difference value threshold of the next threshold adjustment period is calculated:

[0100] T=Bx(1+axM);

[0101] Wherein, T represents the dynamic difference value threshold of the next threshold adjustment period; B represents the basic difference value threshold; a represents the adjustment coefficient, and the value is 0.5-1.0; M represents the average absolute deviation.

[0102] It should be noted that the data difference sequence in the current threshold adjustment period is called from the cache area and the storage area.

[0103] The threshold adjustment period in the technical solution is determined according to the number of data sampling, for example, 30 times of data sampling is one threshold adjustment period.

[0104] In the current threshold adjustment period, the data difference sequence written in the current threshold adjustment period is called, and the absolute value is taken to form a data difference absolute value sequence.

[0105] The above difference absolute value sequence is counted, and the mean absolute deviation M is calculated, which reflects the overall fluctuation degree of the sensor data change in the threshold adjustment period.

[0106] According to the mean absolute deviation, the base difference threshold and the adjustment coefficient, the dynamic difference threshold of the next threshold adjustment period is calculated, and through the above-mentioned manner, the local processor can adaptively adjust the dynamic difference threshold of the next threshold adjustment period according to the actual situation of the environmental data change amplitude at the end of each threshold adjustment period, so that the data judgment strategy is more in line with the dynamic environmental characteristics.

[0107] When the sensor environment fluctuates greatly, the mean absolute deviation M increases, and the dynamic difference threshold is adjusted upward, thereby reducing the frequent writing caused by noise or slight disturbance and reducing the storage burden and the Flash write amplification effect.

[0108] When the environment is relatively stable, the mean absolute deviation M is small, and the dynamic difference threshold is automatically adjusted downward, so that the system can capture the slight but valuable data change and improve the data fidelity.

[0109] Through the dynamic threshold adjustment mechanism, the problems of “too sensitive” or “too dull” under the traditional fixed threshold strategy are avoided, and the balance between storage efficiency and data accuracy is realized.

[0110] Further, the method further comprises:

[0111] determining whether the dynamic difference threshold of the next threshold adjustment period is within a set dynamic difference threshold interval;

[0112] If the dynamic difference threshold of the next threshold adjustment period is greater than the set maximum dynamic difference threshold, the dynamic difference threshold of the next threshold adjustment period is adjusted to the set maximum dynamic difference threshold.

[0113] If the dynamic difference threshold of the next threshold adjustment period is less than the set minimum dynamic difference threshold, the dynamic difference threshold of the next threshold adjustment period is adjusted to the set minimum dynamic difference threshold.

[0114] By setting the upper and lower limits of the threshold interval, the dynamic difference threshold is effectively prevented from excessive drift in extreme cases. For example, when the environmental data fluctuates sharply, the dynamic difference threshold is limited within the maximum value to avoid complete loss of small changes due to the excessively high threshold; when the environment is extremely stable, the dynamic difference threshold is limited above the minimum value to avoid excessive sensitivity to weak noise due to the excessively low threshold.

[0115] The interval constraint mechanism improves the stability and controllability of the dynamic difference threshold updating process, ensuring that the system can maintain reasonable compression efficiency and data fidelity under different working conditions during long-term operation.

[0116] It should be noted that the method further comprises:

[0117] Upon entering the current day, the daytime basic difference threshold and the nighttime basic difference threshold of the current day are set according to the current seasonal information and the current location information.

[0118] It should be noted that the basic difference threshold is also not a fixed value, but is dynamically set according to the current seasonal information (spring, summer, autumn, winter) and the current location information (e.g. Xinjiang and Guangdong). For example, in spring and summer, the basic difference threshold can be appropriately increased due to the larger temperature fluctuation range; while in autumn and winter, the basic difference threshold can be appropriately decreased due to the smaller overall temperature fluctuation; for example, in Xinjiang and other regions with significant diurnal temperature difference, a larger day-night threshold difference can be set; while in Guangdong and other regions with smaller temperature difference, a smaller day-night threshold difference can be set.

[0119] Moreover, in the present technical solution, the daytime basic difference threshold and the nighttime basic difference threshold of the current day are set, which can set a lower basic difference threshold during the deep night period when the temperature fluctuation is smaller, and a higher basic difference threshold during the daytime temperature active period, making it more consistent with the laws of the physical world.

[0120] By introducing the setting of "daytime basic difference threshold" and "nighttime basic difference threshold", and combining seasonal and geographical location for adaptive adjustment, the present application can better adapt to the environmental change laws of the physical world, making the adjustment of dynamic threshold more reasonable. This solution further improves the compression efficiency and data fidelity of the system under different times and environments, achieving intelligent and fine management in long-term operation.

[0121] Embodiment Four

[0122] It should be noted that the method further comprises:

[0123] controlling the sensor to sample data based on the initial sampling interval;

[0124] If the current data difference is not greater than the dynamic difference threshold, and the difference change rate is not greater than the change rate threshold, the current sampling data is regarded as invalid data, and the number of invalid data with continuity is recorded by the counter;

[0125] When the value recorded in the counter reaches the low-power triggering threshold, the initial sampling interval is multiplied by the interval coefficient, and the sensor is controlled to sample data based on the updated sampling interval; the interval coefficient is greater than 1.

[0126] In this embodiment, a counter is maintained to record the number of times that the sampling data is continuously skipped (i.e., invalid data is not written). When the count of the counter reaches the low-power triggering threshold (for example, 10 consecutive skips), the system determines that the current environmental data is stable, and automatically multiplies the initial sampling interval by the interval coefficient. For example, if the interval coefficient is 2, the initial sampling interval is doubled (for example, from 1 minute to 2 minutes, and then to 4 minutes).

[0127] By dynamically extending the sampling interval, the technical solution can significantly reduce the number of samplings during a period of time when the data does not change significantly, thereby reducing the energy consumption of MCU operation and sensor operation, and prolonging the battery life.

[0128] The use of the "continuous invalid data triggering" mechanism enables the adjustment of the sampling interval to be based on the actual data change, rather than simply relying on a fixed strategy, thereby ensuring that the power consumption is reduced during the stable stage, while not missing sudden data changes.

[0129] In combination with the dynamic threshold determination mechanism, the technical solution realizes double low-power control: on the one hand, the number of Flash write operations is reduced, and on the other hand, the sampling and calculation frequency is reduced, thereby enabling the system to operate stably for a long time under low computing power and low energy consumption conditions.

[0130] Further, the method further comprises:

[0131] If the updated sampling interval exceeds the preset maximum sampling interval, the update of the sampling interval is cancelled.

[0132] Specifically, after the local processor detects that the number of consecutive invalid data in the counter reaches the low-power triggering threshold, the current sampling interval is enlarged by the interval coefficient to realize low-power sampling. However, to avoid the case that the sampling interval is too large, causing the data collection to be too sparse, and thus important change points are missed, the application performs the following judgment before updating the sampling interval: if the updated sampling interval is greater than the preset maximum sampling interval, the update operation is cancelled, and the sampling interval is maintained as the maximum sampling interval; if the updated sampling interval is less than or equal to the maximum sampling interval, the sampling interval update takes effect.

[0133] The maximum sampling interval can be set according to the application scenario. For example, in the temperature and humidity monitoring scenario, the maximum sampling interval can be set to several minutes to ensure that the environment mutation can still be captured in time; in the pressure or vibration monitoring scenario, the maximum sampling interval can be set to seconds to ensure that key events will not be missed.

[0134] By setting a maximum upper limit to the sampling interval, the application avoids the risk of data integrity caused by unlimited interval amplification in the low-power mode, and ensures the time sequence continuity of the data.

[0135] Further, the method further comprises:

[0136] If the current sampling data is written into the cache area, the record of the counter is cleared.

[0137] Specifically, when the local processor determines that a certain sampling data is valid based on the difference and the change rate, the following operations are performed:

[0138] The valid data is written into the cache area, and the bound difference and sampling interval information are also written;

[0139] The counter for recording the number of "continuous invalid data" is cleared;

[0140] In the subsequent sampling process, the sensor sampling is controlled to start from the initial sampling interval again, avoiding that the sampling interval remains too large after a long time.

[0141] Through the above mechanism, the technical solution can automatically restore the sampling sensitivity when new valid data is monitored, to ensure that sudden or periodic changes will not be ignored due to the previous low-power sampling strategy.

[0142] By clearing the counter and restoring the sampling interval when valid data appears, the technical solution ensures the quick response capability of the system to new changes, and will not cause slow reaction due to the low-power mode.

[0143] This mechanism, combined with the "interval amplification" strategy and the "maximum interval protection" strategy in the embodiment, constitutes a complete low-power adaptive closed loop: when the data is stable, the sampling interval is gradually extended to save energy, and when the data fluctuates, the sampling frequency is quickly restored to ensure accuracy.

[0144] Embodiment five

[0145] It should be noted that the preset storage condition at least includes:

[0146] The current data difference is greater than the product of the dynamic difference threshold and the storage coefficient, and / or the difference change rate is greater than the product of the change rate threshold and the storage coefficient;

[0147] The set storage time is reached;

[0148] the buffer usage rate exceeds a preset buffer threshold;

[0149] the system is about to enter a deep sleep mode.

[0150] When the current data difference is greater than the product of the dynamic difference threshold and the storage coefficient, and / or the difference rate of change is greater than the product of the change rate threshold and the storage coefficient, the write operation is triggered. This mechanism ensures that when the data changes significantly, it can be recorded immediately. In specific practice, the storage coefficient can be 1.5, that is, when the current data difference is greater than 1.5 times the dynamic difference threshold, and / or the difference rate of change is greater than 1.5 times the change rate threshold, the preset storage condition is triggered.

[0151] When the buffer running time reaches the set storage time threshold, even if the data fluctuation is not large, the write operation is also executed to ensure the time continuity of long-term monitoring data.

[0152] When the buffer usage rate exceeds the preset buffer threshold, the write operation is triggered to prevent data loss caused by buffer overflow. In specific practice, the preset buffer threshold is 80%.

[0153] When the system is about to enter a deep sleep mode, the write operation is triggered to ensure that the data in the cache is stored on disk before sleep.

[0154] Through multi-condition constraints, the present application avoids the problems of not timely storage or too frequent storage caused by a single trigger strategy, and achieves a balance between safety, real-time performance and efficiency of the storage operation.

[0155] Embodiment six

[0156] It should be noted that the method further comprises:

[0157] The total number of collected data, the number of valid data and the number of write operations in the storage area are counted as local statistical indicators.

[0158] The local statistical indicators are sent to the upper computer so that the upper computer calculates the proportion of valid data, the number and proportion of invalid data, the average number of data written each time in the storage area, the original storage estimate, the storage estimate after compression and the storage space saving ratio according to the local statistical indicators.

[0159] Specifically, the local processor counts the following indicators during operation:

[0160] The total number of collected data, the number of valid data and the number of write operations in the storage area are counted.

[0161] Then the local statistical indicators are sent to the upper computer to make the upper computer calculate the valid data proportion, invalid data quantity and proportion, average data quantity per write in the storage area, original storage estimation, compressed storage estimation and storage space saving proportion according to the local statistical indicators.

[0162] The indicators are described as follows:

[0163] Total sampling data quantity: indicates the total data points collected by the sensor in a specified running period;

[0164] Valid data quantity and proportion: records the data points meeting the write-in condition and entering the cache area and the proportion in the total sampling quantity;

[0165] Invalid data quantity and proportion: records the data quantity judged as invalid and not written in and the proportion;

[0166] Storage area write-in times: indicates the actual write-in operation times of the Flash or EEPROM;

[0167] Average data quantity per write in the storage area: reflects the batch efficiency of a single write-in operation;

[0168] Original storage estimation: the storage space estimation value required based on the original piece-by-piece storage strategy;

[0169] Compressed storage estimation: the actual storage space estimation value based on the compression method of the application;

[0170] Storage space saving proportion: calculated by comparing the original storage estimation and the compressed storage estimation.

[0171] In the technical solution, the local MCU only needs to count the total sampling data quantity, valid data quantity and storage area write-in times, and the remaining indicators can be calculated by the upper computer, thereby saving the storage resources of the local MCU.

[0172] After obtaining the above indicators, the upper computer further jointly reports the above indicators and the reconstruction mean square error obtained when linearly interpolating and reconstructing the compressed data. Through this way, the developer can not only understand the optimization effect of the system in storage and power consumption, but also evaluate the accuracy performance of data reconstruction.

[0173] The experimental data of the technical solution are as follows:

[0174] The experiment simulates three possible situations in a period of time in one day: normal temperature rise and fall, sudden temperature drop or rise, and almost unchanged temperature fluctuation.

[0175] LOG log (sampling interval is 30 min):

[0176] Time 00:00: 21.8°C -> [store] (initial value)

[0177] Time 00:30: 21.6°C (delta = -2) -> [skip]

[0178] Time 01:00: 21.4°C (delta = -4) -> [skip]

[0179] Time 01:30: 21.3°C (delta = -5) -> [store]

[0180] Time 02:00: 21.1°C (delta = -2) -> [skip]

[0181] Time 02:30: 21.0°C (delta = -3) -> [store]

[0182] Time 03:00: 20.9°C (delta = -1) -> [skip]

[0183] Time 03:30: 20.8°C (delta = -2) -> [skip]

[0184] Time 04:00: 20.8°C (delta = -2) -> [skip]

[0185] >> System stable, enter low power mode, sample interval adjusted to 1.0 hours

[0186] >> Write 3 records to Flash, time: 05:00, reason: data significant change

[0187] Time 05:00: 20.6°C (delta = -4) -> [store]

[0188] >> Detected valid change, resume high frequency sampling, interval adjusted to 0.5 hours

[0189] Time 05:30: 20.6°C (delta = +0) -> [skip]

[0190] >> Write 1 record to Flash, time: 06:00, reason: data significant change

[0191] Time 06:00: 20.0°C (delta = -6) -> [store]

[0192] Time 06:30: 20.1°C (delta = +1) -> [skip]

[0193] Time 07:00: 20.2°C (delta = +2) -> [skip]

[0194] >> Write 1 record to Flash, time: 07:30 Reason: data significant change

[0195] Time 07:30: 20.6°C (Delta = +6) -> [Store]

[0196] Time 08:00: 21.0°C (Delta = +4) -> [Skip]

[0197] Time 08:30: 21.5°C (Delta = +9) -> [Store]

[0198] Time 09:00: 22.2°C (Delta = +7) -> [Store]

[0199] Time 09:30: 23.0°C (Delta = +8) -> [Store]

[0200] Time 10:00: 23.9°C (Delta = +9) -> [Store]

[0201] Time 10:30: 24.8°C (Delta = +9) -> [Store]

[0202] Time 11:00: 25.7°C (Delta = +9) -> [Store]

[0203] Time 11:30: 26.6°C (Delta = +9) -> [Store]

[0204] Time 12:00: 27.4°C (Delta = +8) -> [Store]

[0205] Time 12:30: 28.1°C (Delta = +7) -> [Store]

[0206] Time 13:00: 28.7°C (Delta = +6) -> [Skip]

[0207] Time 13:30: 29.0°C (Delta = +9) -> [Store]

[0208] Time 14:00: 29.1°C (Delta = +1) -> [Skip]

[0209] >> Write 11 records to Flash, time: 14:30 Reason: data significant change

[0210] Time 14:30: 26.1°C (Delta = -29) -> [Store]

[0211] Time 15:00: 25.4°C (Δ=-7) →

store

[0212] Time 15:30: 24.7°C (Δ=-7) →

skip

[0213] Time 16:00: 24.2°C (Δ=-12)→

store

[0214] Time 16:30: 23.8°C (Δ=-4) →

skip

[0215] Time 17:00: 23.5°C (Δ=-7) →

skip

[0216] Time 17:30: 23.2°C (Δ=-10)→

skip

[0217] >> System stable, enter low-power mode, sampling interval adjusted to 1.0 hours

[0218] Time 18:30: 23.2°C (Δ=-10)→

store

[0219] >> Detected significant change, resume high-frequency sampling, interval adjusted to 0.5 hours

[0220] Time 19:00: 23.1°C (Δ=-1) →

skip

[0221] Time 19:30: 23.0°C (Δ=-2) →

skip

[0222] Time 20:00: 22.9°C (Δ=-3) →

skip

[0223] >> System stable, enter low-power mode, sampling interval adjusted to 1.0 hours

[0224] Time 21:00: 22.7°C (Δ=-5) →

skip

[0225] Time 22:00: 22.5°C (Δ=-7) →

skip

[0226] Time 23:00: 22.3°C (Δ=-9) →

store

[0227] >> Detected significant change, resume high-frequency sampling, interval adjusted to 0.5 hours

[0228] Time 23:30: 22.2°C (Δ=-1) →

skip

[0229] >> Write 4 records to Flash, time: 23:30, reason: reach the set storage time.

[0230] From the log data, it can be seen that the system works normally, and different basic thresholds are set at different times. The dynamic threshold is also updated in real time. The dynamic sampling frequency module correctly triggers the low-power mode when the temperature change is not large, and restores the high-frequency sampling when there is a valid change.

[0231] Figure 2 A comparison chart of valid data (red dots), invalid data (gray dots), and interpolation data reconstructed by the host computer (blue dots).

[0232] From the above table 1, it can be seen that the technical scheme can significantly save storage space, and the compression rate of the sampling data is 50%~80%. Figure 2 It can be seen that in the first half 6:00-14:00, the system accurately captures the temperature rise without missing key temperature data.

[0233] The middle part 14:30 has a sudden temperature drop (heavy rain, greenhouse water spraying, etc.), and the system also accurately captures the data change, indicating that the technical scheme has strong robustness. Although the accuracy of the reconstructed interpolation data is slightly lower when facing sudden changes, it can correctly reflect the trend and has little impact on future analysis.

[0234] The latter part 19:00-6:00 the next day after sunset, the temperature slowly decreases, and the system only records the key nodes. And the temperature simulated by the linear interpolation method after the fact can basically reflect the skipped data.

[0235] The compression performance report of this simulation (30-minute accuracy) is as follows:

[0236] Table 1 Compression Performance Report

[0237]

[0238] From the above table 1, it can be seen that the technical scheme can significantly save storage space, and the compression rate of the sampling data is 50%~80%.

[0239] In summary, in the technical scheme, the difference between the sensor sampling data and the last valid write data and the change rate of the difference are jointly determined, and only when the dynamic difference threshold or the change rate threshold is exceeded, the cache is written, effectively avoiding the waste of space caused by storing each original data. Compared with directly storing the complete timestamp and floating point value of each record, the technical scheme can significantly reduce the amount of stored data, greatly improve the utilization rate of limited storage resources, and thus prolong the local storage period.

[0240] The effective data is temporarily stored in the cache area, and is uniformly written into the storage area when the preset storage condition is triggered, so that batch writing operation is realized. This mode avoids the write amplification effect caused by frequent writing triggered by slight changes, reduces the erase-write frequency of the storage area, and significantly prolongs the service life of the memory device.

[0241] The dynamic difference threshold changing over time is introduced, which can be adaptively adjusted in combination with different environmental states. When the environmental data fluctuates greatly, the threshold is raised to avoid invalid writing; when the environment is stable, the threshold is lowered to improve the sensitivity to slight effective changes, so as to balance between compression efficiency and data fidelity.

[0242] The compressed data is reconstructed by the host computer using linear interpolation method, which can greatly reduce the storage and writing times while ensuring the continuity and trend integrity of the data sequence, taking into account the storage efficiency and data availability. At the same time, the local device is only responsible for reporting the compressed original data packet, thereby maximizing the saving of local computing resources and wireless communication bandwidth. This architecture also makes the future upgrade of the reconstruction algorithm unnecessary to change the local firmware, which has high maintainability.

[0243] Embodiment seven

[0244] Figure 3 is a structural schematic diagram of a sensor data compression storage system provided by an embodiment of the present application, referring to Figure 3 A sensor data compression storage system comprises:

[0245] The local processor 21 and the host computer 22;

[0246] The local processor 21 and the host computer 22 are connected through wireless communication:

[0247] The local processor 21 is configured to execute the sensor data compression storage method in any one of the above embodiments.

[0248] It can be understood that the same or similar parts in the above embodiments can be mutually referred to, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0249] It should be noted that in the description of the present application, the terms "first", "second" and the like are only for the purpose of description, and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "a plurality of" is at least two.

[0250] Any processes or methods described in the flowcharts or elsewhere in this specification can be understood as representing one or more modules, segments, or portions of code that includes executable instructions for performing specific logical functions or steps, and the various embodiments of the application can include additional or fewer steps performing the same or equivalent functions in the same or equivalent order as those described and shown in the figures. The various embodiments of the application can be implemented in software, hardware, firmware, or a combination thereof.

[0251] It should be understood that portions of the application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0252] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiments can be completed by programs instructing related hardware, and the programs can be stored in a computer readable storage medium, and when executed, include one or a combination of steps of the method embodiments.

[0253] In addition, each functional unit in each embodiment of the present application can be integrated into one processing module, or each unit can be physically present alone, or two or more units can be integrated into one module. The above-mentioned integrated module can be realized in the form of hardware or in the form of a software function module. The integrated module, if realized in the form of a software function module and sold or used as an independent product, can also be stored in a computer readable storage medium.

[0254] The above-mentioned storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0255] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0256] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that those skilled in the art can make changes, modifications, substitutions and variations to the above-described embodiments within the scope of the present application.

Claims

1. A method for compressing and storing sensor data, characterized in that, include: The sensor is controlled to sample data based on a set sampling interval, and the current sampled data input from the sensor is received. Calculate the difference between the current sampled data and the previous valid sampled data, and record it as the current data difference; Retrieve the written data difference sequence and calculate the rate of change of the current data difference; Determine whether the current data difference is greater than a dynamic difference threshold, and whether the rate of change of the difference is greater than a rate of change threshold; wherein, the dynamic difference threshold is dynamically adjusted after data sampling at a preset interval; If the current data difference is greater than the dynamic difference threshold and the rate of change of the difference is greater than the rate of change threshold, the current sampled data is considered valid sampled data, and the current sampled data, the current data difference, and the current sampling interval are written to the buffer; otherwise, the current sampled data is considered invalid sampled data and discarded. If the current sampled data is the first sampled data of the current day, it is directly regarded as valid sampled data, and the timestamp of the current sampled data is additionally written; Determine whether the previous valid sample data is the first sample data of the current day; If the previous valid sample data is not the first sample data of the current day, the previous valid sample data will be erased; if the previous valid sample data is the first sample data of the current day, it will not be erased. When the preset storage conditions are triggered, the cached data in the cache area is written to the storage area to form compressed data; The system receives an upload command from the host computer and uploads the compressed data from the storage area to the host computer; the compressed data is then reconstructed in the host computer using linear interpolation. After sampling data a preset number of times at each interval, the dynamic difference threshold is dynamically adjusted, including: Using a preset number of data samplings as a threshold adjustment period, retrieve the data difference sequence within the current threshold adjustment period and construct a data difference absolute value sequence; Calculate the mean absolute deviation of the sequence of absolute values ​​of data differences; Calculate the dynamic difference threshold for the next threshold adjustment cycle based on the mean absolute deviation, the baseline difference threshold, and the adjustment coefficient: T = B × (1 + a × M); Where T represents the dynamic difference threshold of the next threshold adjustment cycle; B represents the basic difference threshold; a represents the adjustment coefficient, which ranges from 0.5 to 1.0; and M represents the mean absolute deviation. Determine whether the dynamic difference threshold for the next threshold adjustment cycle is within the set dynamic difference threshold range; If the dynamic difference threshold of the next threshold adjustment period is greater than the set maximum dynamic difference threshold, then the dynamic difference threshold of the next threshold adjustment period will be adjusted to the set maximum dynamic difference threshold. If the dynamic difference threshold of the next threshold adjustment period is less than the set minimum dynamic difference threshold, then the dynamic difference threshold of the next threshold adjustment period will be adjusted to the set minimum dynamic difference threshold. The method further includes: The sensor is controlled to sample data based on the initial sampling interval; The number of consecutive invalid data points is recorded using a counter. When the value recorded in the counter reaches the low-power trigger threshold, the initial sampling interval is multiplied by an interval coefficient, and the sensor is controlled to sample data based on the updated sampling interval; the interval coefficient is greater than 1.

2. The method according to claim 1, characterized in that, The method further includes: Upon entering the current day, set the daytime basic difference threshold and nighttime basic difference threshold based on the current season information and current location information.

3. The method according to claim 1, characterized in that, The method further includes: If the updated sampling interval exceeds the preset maximum sampling interval, then the update of the current sampling interval is cancelled.

4. The method according to claim 1, characterized in that, The method further includes: If the current sampled data is written to the buffer, the counter record will be cleared.

5. The method according to claim 1, characterized in that, The preset storage conditions include at least the following: The current data difference is greater than the product of the dynamic difference threshold and the storage coefficient, and / or the rate of change of the difference is greater than the product of the rate of change threshold and the storage coefficient; The set storage time has been reached; The buffer usage exceeds the preset cache threshold; The system is about to enter deep sleep mode.

6. The method according to claim 1, characterized in that, The method further includes: The total number of collected data, the number of valid data, and the number of writes to the storage area are used as local statistical indicators. The local statistical indicators are sent to the host computer so that the host computer can calculate the percentage of valid data, the number and percentage of invalid data, the average amount of data written to the storage area each time, the original storage estimate, the compressed storage estimate, and the storage space saving ratio based on the local statistical indicators.

7. A sensor data compression and storage system, characterized in that, include: Local processor and host computer; The local processor is connected to the host computer via wireless communication. The local processor is used to execute a sensor data compression and storage method according to any one of claims 1-6.

Images