An internet of things big data intelligent acquisition terminal
Patent Information
- Application Number
- CN202610406846.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-31
- Publication Date
- 2026-09-29
AI Technical Summary
[0006]针对现有技术的不足，本发明提供了一种物联网大数据智能采集终端，解决了现有技术难以兼顾数据价值调度与网络适配，以此难以满足精准高效采集需求的问题
[0016]（1）、该物联网大数据智能采集终端，通过数据采集与分段单元与数据评估单元配合，实现对传感器时序数据的精细化价值区分，先采集多源传感器时序数据流，完成预处理后，按预设时间窗口划分成规整数据段，避免零散数据处理混乱，并针对每段数据做深度波动分析，提取对应的评估集，用以此综合算出数据价值度，从而能精准揪出包含突发变化、偏离常规走势的关键数据，与平稳冗余数据区分开，既不会漏掉高价值数据的关键信息，也能减少无效数据占用终端处理资源，还能减轻后续云端数据分析压力，更贴合现场使用需求。
Smart Images
Figure CN122845668A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of Internet of Things (IoT) data processing technology, specifically to an IoT big data intelligent acquisition terminal. Background Technology
[0002] As an important component of the new generation of information technology, the Internet of Things (IoT) is profoundly changing many fields. With the continuous advancement of sensor technology and the continuous reduction of deployment costs, various sensing devices are widely distributed in the physical space, forming the nerve endings of the IoT system. These sensors continuously collect various physical quantities, providing raw data support for upper-layer applications.
[0003] In industrial IoT scenarios, key equipment typically deploys multiple sensor nodes, each sampling at a frequency of hundreds or even thousands of times per second, forming a continuous data stream. This massive amount of time-series data contains a wealth of information.
[0004] Furthermore, with the rise of edge computing technology, IoT data acquisition terminals are gradually evolving from simple data forwarding devices into intelligent nodes with local processing capabilities. By integrating data processing algorithms into the acquisition terminal, analysis can be performed at the data source, effectively alleviating cloud computing pressure and network transmission burden.
[0005] Based on the above findings, the limitations of existing technologies include at least the following problems: When processing multi-source sensor time-series data, existing technologies struggle to differentiate the priority of data components by considering the inherent characteristics of sensor data, such as fluctuations and abrupt changes. They also fail to synchronously match network conditions for adaptive scheduling. This can easily lead to high-value critical data being over-compressed and distorted, while low-value redundant data consumes a large amount of transmission resources. Furthermore, network link fluctuations can easily cause transmission delays, data packet loss, or low transmission efficiency, making it difficult to balance data transmission integrity with network resource utilization. Consequently, these technologies are ill-suited to meet the precise and efficient requirements of big data collection under complex operating conditions. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides an IoT big data intelligent data acquisition terminal, which solves the problem that existing technologies struggle to balance data value scheduling and network adaptation, thus failing to meet the needs for accurate and efficient data acquisition.
[0007] To achieve the above objectives, the present invention provides the following technical solution: an IoT big data intelligent acquisition terminal, comprising: a data acquisition and segmentation unit for acquiring time-series data streams from multiple sources of sensors and segmenting them to form several data segments; a data evaluation unit for performing fluctuation analysis on the data segments to determine the data value of each data segment; a compression scheduling evaluation unit for acquiring network status data and performing comprehensive processing in conjunction with the data value to obtain a compression scheduling evaluation value for each data segment; and a differential processing unit for performing differential compression and uploading processing on the data segments based on the compression scheduling evaluation value.
[0008] Furthermore, the specific steps for forming several data segments are as follows: preprocessing the time-series data stream; and dividing the preprocessing result based on a preset time window to form several data segments.
[0009] Furthermore, the specific steps for determining the data value of each data segment are as follows: Based on the data segment, extract its corresponding evaluation set, including fluctuation entropy value, mutation factor and trend deviation; perform comprehensive processing on the evaluation set to determine its corresponding data value.
[0010] Further, the specific steps for extracting the evaluation set for each data segment are as follows: perform probability distribution statistical processing on the data segment to obtain its corresponding fluctuation entropy value; perform differential detection processing on the data segment based on a preset mutation threshold to obtain its corresponding mutation factor; and perform fitting and comparison processing on the data segment based on a preset baseline curve to obtain its corresponding trend deviation.
[0011] Furthermore, the network status data includes signal strength, uplink queue backlog, network type, network connection status, and signal-to-noise ratio. The specific steps to obtain the compression scheduling evaluation value for each data segment are as follows: read the network status data and analyze the network quality evaluation value; fuse the network quality evaluation value with the data value to obtain its corresponding compression scheduling evaluation value.
[0012] Further, the specific steps for analyzing network quality assessment values are as follows: Based on the network state data, a network state assessment set is constructed, including channel quality assessment values, link assessment values, and congestion assessment values; the network state assessment set is then comprehensively processed to obtain the network quality assessment values.
[0013] Furthermore, the specific steps for constructing the network state assessment set are as follows: jointly mining the signal strength and the signal-to-noise ratio to extract the channel quality assessment value; performing discrimination and level mapping processing on the network standard and the network connection state to extract the link assessment value; and mapping the uplink queue backlog number to extract the congestion assessment value.
[0014] Furthermore, the specific steps for differential compression and uploading of the data segment are as follows: comparing the compression scheduling evaluation value with the preset compression scheduling evaluation interval; and taking corresponding compression and uploading measures for the data segment based on the comparison result.
[0015] The present invention has the following beneficial effects:
[0016] (1) This IoT big data intelligent acquisition terminal, through the cooperation of the data acquisition and segmentation unit and the data evaluation unit, realizes the fine value differentiation of sensor time series data. First, it collects multi-source sensor time series data streams, completes preprocessing, and divides them into regular data segments according to the preset time window to avoid the chaos of fragmented data processing. It also performs in-depth fluctuation analysis on each data segment, extracts the corresponding evaluation set, and uses it to comprehensively calculate the data value. In this way, it can accurately identify key data containing sudden changes and deviations from the normal trend, and distinguish them from stable and redundant data. It will not miss the key information of high-value data, reduce the use of invalid data to occupy terminal processing resources, and reduce the pressure of subsequent cloud data analysis, which is more in line with the needs of on-site use.
[0017] (2) The IoT big data intelligent acquisition terminal obtains network status data through the compression scheduling evaluation unit, integrates and analyzes it to obtain a network status evaluation set, and analyzes the network quality evaluation value based on it. Then, it merges the data value to obtain the compression scheduling evaluation value of each data segment. The differentiation processing unit compares the evaluation value with the preset interval and matches the corresponding compression upload scheme. This can effectively avoid the lag caused by network fluctuations, reasonably allocate bandwidth resources, prevent low-value data from crowding the transmission space, and ensure that high-value data is transmitted stably with priority, so as to adapt to the IoT big data acquisition and transmission needs under various complex working conditions.
[0018] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0019] Figure 1 This is a block diagram of an IoT big data intelligent acquisition terminal according to the present invention.
[0020] Figure 2 This is a flowchart illustrating the specific steps involved in obtaining the compression scheduling evaluation value of each data segment in an IoT big data intelligent acquisition terminal according to the present invention. Detailed Implementation
[0021] Please see Figure 1This invention provides a technical solution: an IoT big data intelligent acquisition terminal, comprising: a data acquisition and segmentation unit for acquiring time-series data streams from multiple sources of sensors and segmenting them to form several data segments (each data segment corresponds to the sampling data of a sensor within a continuous time period); a data evaluation unit for performing fluctuation analysis on the data segments to determine the data value of each data segment; a compression scheduling evaluation unit for (after the data segments are segmented) acquiring network status data and performing comprehensive processing in conjunction with the data value to obtain a compression scheduling evaluation value for each data segment; and a differentiated processing unit for performing differentiated compression and uploading processing on the data segments based on the compression scheduling evaluation value.
[0022] The specific steps for differential compression and uploading of data segments are as follows: The compression scheduling evaluation value is compared with the preset compression scheduling evaluation interval; based on the comparison result, corresponding compression and uploading measures are taken for the data segments, specifically:
[0023] If the compression scheduling evaluation value is lower than the preset lower limit of the compression scheduling evaluation interval, then only the basic statistical features of the data segment are extracted as representatives, the original data points are discarded, and the feature data is uploaded when the network is idle.
[0024] If the compression scheduling evaluation value is within the preset compression scheduling evaluation range, then the data segment is subjected to lossy compression with a medium compression rate, retaining key feature points, and is uploaded in order to the normal sending queue.
[0025] If the compression scheduling evaluation value is higher than the upper limit of the preset compression scheduling evaluation interval, then the data segment will be compressed using a lossless compression algorithm to fully preserve all sampling points and will be immediately sent to the transmission queue for priority uploading.
[0026] Specifically, the steps for forming several data segments are as follows: Preprocessing the time-series data stream involves receiving raw time-series data acquired in real-time from multiple sensors, using median filtering to remove outliers caused by sensor disturbances or sudden interference. Specifically, the median of each sampling point and the two points before and after it (a total of five points) is used to replace the original value. Then, linear interpolation is used to fill in data gaps caused by communication interruptions. Specifically, the mean of the two valid sampling points before and after the missing point is calculated as the filling value. Finally, the data from each sensor is normalized to the [0, 1] interval according to its respective range.
[0027] The preprocessing results are divided into several data segments based on a preset time window. Specifically, a time window of fixed length of 1 second is set, and each sensor data stream after preprocessing is slidably segmented. Adjacent windows overlap by 0.5 seconds to ensure the continuity of edge data. Each segmented data segment contains all the sampling points of the sensor within the corresponding time window. Finally, a sensor identifier, start timestamp, and end timestamp are added to the data segment.
[0028] In this implementation scheme, a combination of preprocessing and sliding window segmentation is used to make subsequent data processing more stable and reliable. First, the raw time-series data from the sensors is received. Median filtering is used to filter out anomalies caused by sensor disturbances or external interference. Then, linear interpolation is used to complete the data missing due to brief communication interruptions. At the same time, the data from different sensors are uniformly normalized so that all types of data participate in the calculation under the same standard. After preprocessing, the processed data is segmented by sliding window to ensure that the data is not broken or lost at the window connection points. Each data segment is accompanied by the corresponding sensor identifier and start and end timestamps for easy traceability, thus providing a stable and reliable data foundation.
[0029] Specifically, the steps to determine the data value of each data segment are as follows: Based on the data segment, extract its corresponding evaluation set, including volatility entropy value, mutation factor and trend deviation;
[0030] The evaluation set is comprehensively processed to determine its corresponding data value. Specifically, the fluctuation entropy value, mutation factor, and trend deviation are normalized to the interval [0, 1] and then weighted and summed according to preset weights, where the fluctuation entropy value has a weight of 0.4, the mutation factor has a weight of 0.4, and the trend deviation has a weight of 0.2. The weighted sum is the data value of the data segment. The higher the value, the greater the amount of information contained in the data segment and the more worthy it is to be completely preserved.
[0031] The specific steps for extracting the evaluation set for each data segment are as follows: Perform probability distribution statistical processing on the data segment to obtain its corresponding fluctuation entropy value. Specifically, divide the amplitude of all sampling points in the data segment into 10 equally spaced intervals, and count the proportion of the number of sampling points in each interval to the total number of points as the probability distribution; then calculate the fluctuation entropy value according to the information entropy formula. The larger the fluctuation entropy value, the more dispersed the numerical distribution and the richer the fluctuation within the data segment.
[0032] Based on a preset mutation threshold, differential detection processing is performed on the data segment to obtain its corresponding mutation factor. Specifically, the absolute value of the first difference between adjacent sampling points within the data segment is calculated, the number of points whose absolute difference exceeds the preset mutation threshold is counted, and this number is divided by the total number of points in the data segment to obtain the mutation point ratio; at the same time, the mean of all absolute difference values exceeding the threshold is calculated as the mutation amplitude; the mutation factor is the product of the mutation point ratio and the normalized mutation amplitude, which characterizes the severity of sudden changes within the data segment.
[0033] It should be noted that the preset steps for the mutation threshold are as follows:
[0034] Collect historical data from each sensor under normal operating conditions. Calculate the statistical characteristics of the differences between adjacent sampling points based on the historical data, including the mean and standard deviation. Use the sum of the mean and the standard deviation of a preset multiple (e.g., 3 for vibration sensors, 2.5 for temperature sensors, 2 for pressure sensors, etc.) as the baseline abrupt change threshold.
[0035] Real-time monitoring of equipment operating status (shutdown, startup, steady state, abnormal) and dynamic adjustment of baseline thresholds based on the current status:
[0036] When the device is running in a steady state, the mutation threshold is taken as the baseline mutation threshold.
[0037] When the device is in a startup or shutdown transient state, the mutation threshold is 1.5 times the baseline mutation threshold to prevent normal transient processes from being misjudged as mutations;
[0038] When the equipment is in an abnormal state, the mutation threshold is set to 0.8 times the baseline mutation threshold to improve sensitivity to early fault characteristics.
[0039] When the equipment is in a stopped state, mutation detection is not performed, and the mutation factor is directly set to 0;
[0040] Based on the preset baseline curve, the data segment is fitted and compared to obtain its corresponding trend deviation. Specifically, the baseline curve model of each sensor is established based on the historical data of the equipment during normal operation and stored locally; the data points in the current data segment are fitted with linear regression to obtain the slope of the fitted line; the absolute value of the difference between the slope and the slope of the baseline curve for the corresponding time period is calculated, normalized to the [0, 1] interval and used as the trend deviation, which reflects the degree of deviation of the equipment's operating state from the historical normal state.
[0041] In this implementation plan, when determining the data value, fluctuation entropy, mutation factor, and trend deviation are extracted from each data segment, and the final result is obtained through a weighted average. When calculating the fluctuation entropy, the amplitude of the sampling point is divided into 10 equally spaced intervals. The result is obtained by statistically analyzing the interval proportions and using the information entropy formula to reflect the dispersion of the data distribution. The mutation factor is obtained through differential detection. The threshold is first calculated based on the historical data of different sensors to obtain a baseline value, and then adaptively adjusted in combination with the device status to avoid misjudgment and highlight abnormal changes. The trend deviation is obtained by comparing the current data fitting slope with the local baseline curve to reflect the deviation of the operating status. After normalizing the three indicators, they are weighted and summed. The higher the data value, the more information the data contains, and the more suitable it is for complete transmission and storage.
[0042] Specifically, such as Figure 2 As shown, the network status data includes signal strength, uplink queue backlog, network type, network connectivity status, and signal-to-noise ratio. The specific steps to obtain the compression scheduling evaluation value for each data segment are as follows:
[0043] Read network status data and analyze network quality assessment values; fuse network quality assessment values and data value scores to obtain corresponding compression scheduling assessment values. Specifically, assign weights of 0.6 and 0.4 to the network quality assessment values and data value scores respectively, and perform weighted summation to obtain the compression scheduling assessment value. This value is located in the range [0, 1]. The higher the value, the more worthwhile it is to use low compression rate and high priority for uploading the data segment under the current network conditions.
[0044] The signal strength is the power value of the currently received signal, which can be obtained by reading the RSRP register.
[0045] The uplink queue backlog is the number of data packets to be sent, which can be obtained by reading the network card driver queue length.
[0046] The network standard is the type of network currently being accessed, which can be obtained by reading the network registration status register.
[0047] The network connection status indicates whether the current link is connected; 0 indicates disconnected and 1 indicates connected. It can be obtained by reading the socket connection status.
[0048] The signal-to-noise ratio (SNR) is the power ratio of the signal to the noise, and it can be obtained by reading the SNR register.
[0049] The specific steps for analyzing network quality assessment values are as follows:
[0050] Based on network state data, a network state assessment set is constructed, including channel quality assessment values, link assessment values, and congestion assessment values.
[0051] The network state assessment set is comprehensively processed to obtain the network quality assessment value. Specifically, the channel quality assessment value, link assessment value, and congestion assessment value are weighted by 0.5, 0.3, and 0.2 respectively, and then summed to obtain the network quality assessment value. This value is located in the interval [0, 1], and the higher the value, the more favorable the current network conditions are for high-quality data transmission.
[0052] It should be noted that during the weighting process, the congestion assessment value is transformed using the reciprocal suppression mapping function f(x)=1 / (1+x).
[0053] The signal strength and signal-to-noise ratio are jointly mined to extract the channel quality assessment value. Specifically, the signal strength value (in dBm) is mapped to the interval [-140, -40] and normalized, and the signal-to-noise ratio (in dB) is mapped to the interval [0, 30] and normalized. The product of the two normalized values is taken as the channel quality assessment value, which comprehensively reflects the quality of the physical layer transmission conditions.
[0054] The network standard and network connection status are identified and mapped to different levels to extract the link evaluation value. Specifically, the network connection status is checked. If it is 0, the link evaluation value is set to 0. If it is 1, the link evaluation value is mapped to different levels according to the network standard: 5G is mapped to 1.0, 4G to 0.8, Wi-Fi to 0.6, and other standards to 0.4. This value reflects the basic transmission capacity of the link.
[0055] The uplink queue backlog is mapped and the congestion assessment value is extracted. Specifically, the uplink queue backlog is read and divided by the preset maximum backlog threshold (set according to the terminal buffer capacity, the default is 100 data packets) to obtain the congestion assessment value. If the result is greater than 1, it is taken as 1.
[0056] In this implementation scheme, network status data is read to construct a network status assessment set. First, the signal strength and signal-to-noise ratio are normalized and multiplied to obtain the channel quality assessment value. Then, a level mapping is performed based on the network connection status and network standard to obtain the link assessment value. At the same time, the congestion assessment value is calculated by the ratio of the uplink queue backlog to the buffer limit, and a mapping function is used for transformation processing. Then, the network quality assessment value is obtained by combining it with preset weights. Finally, the network quality assessment value is weighted and fused with the data value to obtain the compression scheduling assessment value. The higher this value, the more suitable the data is for uploading in a low-compression, high-priority manner in the current network environment, providing a reliable basis for subsequent differentiated transmission.
[0057] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0058] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. An IoT big data intelligent acquisition terminal, characterized in that, include: The data acquisition and segmentation unit is used to acquire time-series data streams from multiple source sensors and segment them into several data segments. A data evaluation unit is used to perform fluctuation analysis on the data segments and determine the data value of each data segment. The compression scheduling evaluation unit is used to acquire network status data, combine it with the data value, and perform comprehensive processing to obtain the compression scheduling evaluation value of each data segment. The differential processing unit is used to perform differential compression and uploading processing on the data segment based on the compression scheduling evaluation value.
2. The IoT big data intelligent acquisition terminal according to claim 1, characterized in that, The specific steps to form several data segments are as follows: The time-series data stream is preprocessed; The preprocessing results are divided into several data segments based on a preset time window.
3. The IoT big data intelligent acquisition terminal according to claim 1, characterized in that, The specific steps for determining the data value of each data segment are as follows: Based on the data segment, extract its corresponding evaluation set, including fluctuation entropy value, mutation factor and trend deviation; The evaluation set is comprehensively processed to determine its corresponding data value.
4. The IoT big data intelligent acquisition terminal according to claim 3, characterized in that, The specific steps for extracting the evaluation set for each data segment are as follows: The data segment is subjected to probability distribution statistical processing to obtain its corresponding fluctuation entropy value; Based on a preset mutation threshold, differential detection processing is performed on the data segment to obtain its corresponding mutation factor; Based on a preset baseline curve, the data segment is fitted and compared to obtain its corresponding trend deviation.
5. The IoT big data intelligent acquisition terminal according to claim 1, characterized in that, The network status data includes signal strength, uplink queue backlog, network type, network connection status, and signal-to-noise ratio. The specific steps to obtain the compression scheduling evaluation value for each data segment are as follows: Read the network status data and analyze the network quality assessment value; The network quality assessment value and the data value are fused together to obtain the corresponding compression scheduling assessment value.
6. The IoT big data intelligent acquisition terminal according to claim 5, characterized in that, The specific steps for analyzing network quality assessment values are as follows: Based on the network status data, a network status assessment set is constructed, including channel quality assessment values, link assessment values, and congestion assessment values. The network state assessment set is processed to obtain the network quality assessment value.
7. The IoT big data intelligent acquisition terminal according to claim 6, characterized in that, The specific steps for constructing the network state evaluation set are as follows: The signal strength and the signal-to-noise ratio are jointly mined to extract the channel quality assessment value; The network type and network connection status are identified and mapped in a graded manner, and the link evaluation value is extracted. The backlog of the uplink queue is mapped to extract the congestion assessment value.
8. The IoT big data intelligent acquisition terminal according to claim 1, characterized in that, The specific steps for performing differentiated compression and uploading on the data segment are as follows: The compression scheduling evaluation value is compared with the preset compression scheduling evaluation range; Based on the comparison results, corresponding compression and uploading measures are taken for the data segment.