Network and service quality double constraint driven data processing method for space-based internet of things

Abstract

This invention proposes a space-based IoT data processing method driven by dual constraints of network and service quality. First, the CPE obtains network status information via a satellite terminal, including maximum network bandwidth, network latency, and packet loss rate, and calculates the maximum data transmission rate (Kbps) that the network can support. The CPE then acquires user-acceptable data error information from the space-based IoT application server; this error information is the data service quality parameter, used to define the upper limit of user requirements for data error. Next, the CPE performs adaptive data compression and transmits the processed data back to the space-based IoT application server. The satellite terminal converts the raw data into an abstract model using a V2M algorithm, and the satellite receives the data and restores the approximate value using an M2V algorithm. During transmission, OVM (Overbounded Data Message) is used to prioritize uploading overbounded data to ensure timely handling of anomalies, while AMM (Aggregated Model Message) is used for batch transmission of common technical solutions.

Description

Technical Field

[0001] This invention focuses on a space-based Internet of Things (IoT) data processing method under the dual constraints of network service quality conditions and data quality requirements. Background Technology

[0002] With the development of satellite communication technology, space-based Internet of Things (IoT) has become a key architecture for realizing multi-domain data collection across air, space, and sea globally, demonstrating irreplaceable value, especially in scenarios where terrestrial networks are difficult to reach, such as oceans, remote areas, and cross-border regions. Satellite terminals, as the end nodes for data collection and transmission, need to directly establish links with constellations such as low Earth orbit (LEO) satellites to acquire IoT sensing data within a specific area and transmit the data back. However, satellite data transmission and processing face two major challenges: firstly, data backhaul capability is constrained by the quality of service of the network link, limiting the maximum data backhaul rate; secondly, IoT data itself has high spatiotemporal redundancy, requiring data compression processing based on the acceptable data quality for users.

[0003] This invention is designed for space-based Internet of Things (IoT) scenarios and addresses the dual constraints of satellite network service quality and IoT business data quality by providing a data processing method driven by both network and business service quality constraints. The method involves using network and business service quality to jointly determine the data processing algorithm, constructing an abstract data model based on data distribution and user accuracy requirements, and then performing data compression processing.

[0004] In a space-based IoT scenario, the following entities are included: 1. Satellite Terminal: The satellite terminal can connect to the network provided by the satellite to achieve network access and support the backhaul of IoT data. 2. Customer Premises Equipment (CPE): The CPE provides a Wi-Fi hotspot to IoT devices, collecting sensing data from multiple IoT devices within range; the CPE provides data compression capabilities to compress the collected sensing data; the CPE connects to the satellite terminal for data backhaul. 3. IoT Sensing Devices: In a space-based IoT scenario, sensing devices do not require any modification; they only need to use devices from traditional terrestrial IoT systems that support Wi-Fi access; the IoT sensing devices collect environmental data within the target area and send the data to the CPE. 4. Space-based IoT Application Server: The space-based IoT application server aggregates the backhauled data and displays it to the user; the space-based IoT application allows users to set acceptable data error rates and sends this parameter to the CPE, which then compresses the data based on these parameters.

[0005] IoT sensing devices acquire environmental data in real time and send the environmental data to the CPE at regular intervals. The CPE uses an adaptive data compression algorithm to compress the data and sends the compressed data back to the space-based IoT application server. The CPE can obtain network service quality information through satellite terminals and obtain user-acceptable data quality information through the space-based IoT application server. Based on the above information, the CPE sets specific parameters for data compression. Summary of the Invention

[0006] This invention addresses the trade-off between fully utilizing space-based Internet of Things (IoT) network resources and ensuring IoT data quality. It employs an adaptive abstract data model for data compression and utilizes different transmission methods based on different message types, thereby reducing data transmission load.

[0007] The overall execution logic of this invention begins with the perception of network and data quality, providing constraints for subsequent data processing. First, the CPE obtains network status information via a satellite terminal, including maximum network bandwidth, network latency, and network packet loss rate. Based on this information, it calculates the maximum data backhaul rate that the network can support. (Kbps); Simultaneously, the CPE obtains user-acceptable data error information from the space-based IoT application server, which is the data service quality parameter. This information is used to define the user's upper limit requirement for data error. Afterwards, the CPE performs adaptive data compression based on this information and sends the processed data back to the space-based IoT application server.

[0008] The adaptive data compression algorithm logic is as follows: First, based on the user's quality of service requirements (i.e., error tolerance – corresponding parameters) Bandwidth limitations – including nominal bandwidth B and the actual usable bandwidth after excluding packet loss and latency. The range of data variation and the data density in the transmission scenario determine the appropriate mode for constructing the abstract data model. If a precision-centric mode is chosen, the error tolerance standard will be fixed first to ensure data precision, and then the minimum number of bits required to meet that precision will be determined. Then, considering bandwidth limitations and data density, the maximum number of bits that the current bandwidth can support is calculated. If the former (minimum number of encoding bits) is less than or equal to the latter (maximum number of encoding bits allowed by bandwidth), the minimum number of encoding bits is used; if the former is greater than the latter, the number of encoding bits is reduced and the error tolerance standard is adjusted accordingly. If a bandwidth-centric mode is chosen, the number of encoding bits that can fully utilize the current bandwidth is first determined based on bandwidth constraints and data density, and then the actual data precision corresponding to that number of encoding bits is calculated. If the actual precision is better than the initial precision requirement, that number of encoding bits is used; if the actual precision does not meet the initial requirement, the number of encoding bits is maintained, and the error tolerance standard is appropriately relaxed. If neither of the above two modes can meet the initial quality of service requirements, the error tolerance standard needs to be further relaxed. Ultimately, this results in the final error tolerance level derived by the algorithm. The satellite terminal converts the raw data into an abstract model using the V2M algorithm, and the satellite then uses the M2V algorithm to reconstruct the approximate value after receiving the data. During transmission, out-of-bounds data is prioritized for uploading using OVM (Out-of-bounds Data Message) to ensure timely handling of anomalies, while AMM (Aggregate Model Message) is used for batch transmission of standard technical solutions. Attached Figure Description

[0009] Figure 1 It refers to the application scenario.

[0010] Figure 2 This is a flowchart of the present invention. Detailed Implementation

[0011] Traditionally, the precision of data types used for data transmission may far exceed the accuracy requirements of users. Based on this, this invention proposes a data model oriented towards user service quality needs.

[0012] Data service quality, using This is used to describe the upper limit of the error acceptable to the user.

[0013] This indicates the range of data variation within a specific application. Data within this range is called "normal data," and data exceeding this range is called "out-of-bounds data." Based on these parameters, the data range is divided into...

[0014] There are several intervals, and the values ​​within each interval are considered as an abstract concept, with the same concept representing the midpoint value of that interval. Each interval has a width less than [missing information]. The error of the approximation within each interval is less than the upper limit of error. Using a... –bit binary code represents these intervals and the two types of out-of-bounds data. Clearly,

[0015] Among them, all-zero encoding and all-one encoding are used to represent data that exceeds the lower bound and the upper bound, respectively.

[0016] To reduce data transmission volume, each satellite terminal converts the raw data into an abstract data model. This algorithm is called environmental value to data model conversion, or V2M for short. The satellite terminal only needs to send the abstract data model to the satellite. After receiving the abstract data model, the satellite converts it into an approximation of the original data. This conversion algorithm is called data model to environmental value conversion, or M2V for short.

[0017] I. Algorithm 1 (V2M: Algorithm for converting environmental values ​​to data models)

[0018] Algorithm 1 is used to convert the original environmental perception values ​​into the corresponding abstract data model encoding, so as to reduce the amount of data transmitted and achieve data compression within the acceptable error range for users.

[0019] Specifically, firstly, based on the lower bound of the environmental data values ​​in the target business scenario... Upper bound of values and the user's acceptable error limit Determine the interval division method within the normal data range. Set the total number of intervals to:

[0020]

[0021] Based on this, calculate the width of each interval:

[0022]

[0023] Furthermore, the required number of bits for encoding to represent all regular intervals and the two types of superboundary states was determined. The two types of out-of-bounds states correspond to data below the lower bound and data above the upper bound, respectively. The number of bits in the encoding can be represented as:

[0024]

[0025] For each piece of raw sensory data to be processed The conversion should be performed according to the following rules:

[0026] 1. Lower bound exceeding the limit: If If the data is below the lower bound of the normal range, it is considered out-of-bounds data and encoded as a string of length [length missing]. The binary code consisting entirely of zeros.

[0027] 2. Case where the upper bound exceeds the limit: If If the data is above the upper limit of the normal range, it will be classified as out-of-bounds data and encoded as a data of length [length missing]. The binary code consisting entirely of 1s.

[0028] 3. Normal interval situation: If lie in If the value is within the specified range, its interval number is determined based on its position relative to the lower bound. The interval number is obtained by dividing the difference between the original value and the lower bound by the interval width and then rounding down.

[0029]

[0030] Then number the intervals. Convert to length of The binary encoding is used as the encoding of the abstract data model corresponding to the original data. .

[0031] Through the above processing, the original environmental values ​​are mapped to a finite-length binary abstract code, where regular data is represented by interval coding and out-of-bounds data is represented by dedicated reserved coding. This method can discretize continuous environmental values ​​under given error tolerance constraints, thereby reducing bit overhead in subsequent transmission processes.

[0032] II. Algorithm 2 (M2V: Inverse transformation algorithm from data model to environmental value)

[0033] Algorithm 2 is used to reconstruct the compressed data at the sending end by encoding the abstract data model back to an approximation of the original environmental values.

[0034] Specifically, the receiver first uses parameters consistent with those of the sender, including the lower bound of environmental data values. Upper bound of values and upper limit of error Calculate the total number of regular intervals:

[0035]

[0036] And further calculate the width of each interval:

[0037]

[0038] At the same time, determine the number of bits in the encoding:

[0039]

[0040] To improve decoding efficiency, the receiving end pre-constructs an array of interval midpoints to store the representative value corresponding to each regular interval. For the... Each interval, and its corresponding approximate value The midpoint of this interval is calculated as follows:

[0041]

[0042] Upon receiving the abstract data model encoding Then, perform the inverse transformation according to the following rules:

[0043] 1. Standard Encoding Case: If the received encoding is neither all zeros nor all one, it indicates that the encoding corresponds to a standard range. In this case, first encode... Convert to the corresponding decimal range number Then, based on this number, the corresponding midpoint value is read from the pre-constructed array of midpoints in the interval. The midpoint value is then used as an approximate reconstruction of the original environmental values.

[0044] 2. Out-of-bounds encoding: If the received encoding is all 0s or all 1s, it indicates that the corresponding data is out of bounds. All 0s indicate that the original data is below the lower bound, and all 1s indicate that the original data is above the upper bound. For this type of data, the receiving end does not approximate recovery based on the midpoint of the interval, but instead processes it in conjunction with the separately transmitted original out-of-bounds value to ensure accurate representation of the abnormal data.

[0045] This algorithm achieves rapid recovery of environmental numerical approximations from abstract data models through a mapping relationship between "encoding index and interval midpoint". Since the midpoints of each interval can be pre-calculated and cached, complex calculations do not need to be repeated after receiving the encoding; reconstruction can be completed simply by looking up a table, thereby reducing decoding computational overhead and improving recovery efficiency.

[0046] In Algorithm 2, the algorithm uses an array to store... The midpoint of each interval is used as an approximation of the values ​​within those intervals. When a new abstract data model is received, this array is directly queried to retrieve the approximate value, thus avoiding the need for approximation calculations each time. When all-zero or all-one encodings are received, the satellite does not perform data conversion because the original values ​​of these out-of-bounds values ​​will also be sent to the satellite.

[0047] Parameters are set based on business service quality requirements.

[0048] In data transmission scenarios, the core challenge remains balancing transmission efficiency (bandwidth utilization) and data accuracy (error). When introducing unstable factors such as network packet loss and latency, it is necessary to reserve bandwidth redundancy (denoted as ). , , The value can be dynamically adjusted according to the network jitter level, such as when the packet loss rate is 5%. (Take a value of 0.05~0.1) to ensure stability during actual transmission. User QoS requirements (error tolerance) Bandwidth limitations ) and the implied range of data changes The data model parameter design will be mapped through two differentiated adaptation modes. Both modes are based on the principle of "adapting as needed and relaxing accuracy if not met", prioritizing accuracy or bandwidth requirements respectively, and both require bandwidth redundancy calculation first.

[0049] Regardless of the adaptation mode used, the nominal bandwidth limit set by the user must be adjusted first. Converted to available effective bandwidth To reserve redundancy for handling packet loss and latency, the specific formula is as follows:

[0050]

[0051] in, This is the bandwidth redundancy factor, and its value needs to be dynamically configured based on the actual network environment and application scenario. It can be determined through actual testing in the target business scenario to ensure it meets business requirements. value.

[0052] For example, we set up three different network conditions and conducted corresponding tests. Value. Fix a transmission rate and adjust the bandwidth limit under different network environments until transmission stabilizes (e.g., packet loss rate less than 0.1%). This method yields the redundancy coefficient. , and obtained when:

[0053] With a latency of 119ms and jitter of 84ms, Take a value of 0.33~0.51;

[0054] With a latency of 131ms and jitter of 92ms, Take a value of 0.38~0.52;

[0055] With a latency of 143ms and jitter of 100ms, Take a value of 0.24 to 0.44.

[0056] 1. Accuracy-based adaptation mode

[0057] An accuracy-based adaptation mode ensures that the user-defined error tolerance is met. With (i.e., accuracy requirements) as the core, within the available effective bandwidth Under the constraints, the actual bandwidth usage should be minimized as much as possible. The specific process is as follows:

[0058] First, fix the error tolerance set by the user. According to the range of data changes Calculate the minimum number of partitions required to divide the data interval. (All possible data values ​​must be covered):

[0059]

[0060] Because it is necessary to distinguish between two types of out-of-bounds states where "data exceeds the normal range" (upper bound out-of-bounds and lower bound out-of-bounds), it is necessary to calculate the minimum number of bits required to support n normal ranges + 2 out-of-bounds states. :

[0061]

[0062] (ceil is the floor function, which ensures that the number of bits in the encoding can cover all states).

[0063] Based on the data density of the transmission scenario (Number of data units to be transmitted per unit time), based on available effective bandwidth. Calculate the maximum number of bits that the current bandwidth can support for coding. (i.e., "maximum effective number of bits")

[0064]

[0065] (floor is a floor function to avoid the actual bandwidth exceeding the effective upper limit).

[0066] If the effective bandwidth can support the accuracy Directly adopt As the final encoding bit length, the data approximation error is strictly controlled at this point. Within, and bandwidth usage To reach the minimum value under the current precision; if the effective bandwidth cannot support the precision. This means that the current bandwidth cannot support the given accuracy and needs to be increased. With bandwidth adaptation: fixed encoding bit width Calculate the maximum number of intervals that can be covered at this time. :

[0067]

[0068] Back-calculation of the maximum allowed interval width:

[0069]

[0070] Adjust the error tolerance to ε_new:

[0071]

[0072] Ultimately As a practical accuracy standard, ensure that it is within the effective bandwidth capacity range (actual bandwidth usage). It aims to closely match the user's initial accuracy requirements while mitigating the risk of packet loss.

[0073] 2. Bandwidth-based adaptation mode

[0074] The core objective of this mode remains to fully utilize the bandwidth resources set by the user, but it must be based on available and effective bandwidth. Fully configured, not nominal bandwidth Based on this, we will improve accuracy as much as possible (reduce error tolerance). The specific process adjustments are as follows:

[0075] First, based on data density and available effective bandwidth The number of encoding bits l is fixed to the "effective maximum number of encoding bits" l_max_eff to ensure that the effective bandwidth is fully utilized:

[0076]

[0077] At this point, the actual total bit rate .

[0078] according to Calculate the maximum number of intervals that can be covered. (Including 2 superboundary states):

[0079]

[0080] Combined with the range of data changes By reverse calculation, the maximum allowed interval width can be determined as follows:

[0081]

[0082] (like Unable to be (Divide by integer and round up to ensure coverage), thus obtaining the minimum error tolerance (highest precision) under the current effective bandwidth:

[0083]

[0084] at this time The error tolerance is less than the user's initial setting, achieving "full utilization of effective bandwidth + exceeding accuracy requirements." This fully utilizes bandwidth resources, mitigates the impact of packet loss / latency through redundancy reservations, and improves data accuracy. If the data variation range... Too large (e.g.) (far exceeding the range that the encoding bits can cover), resulting in Still greater than the user's initial accuracy requirement, at this point maintain Unchanged (continue to fully utilize available bandwidth, without wasting resources), will As the final accuracy standard, by relaxing the accuracy, we ensure that data transmission is still feasible under the constraints of network packet loss / latency, and that effective bandwidth is fully utilized (avoiding redundant bandwidth idleness).

[0085] Application layer data transmission mechanism

[0086] To ensure timely transmission of outliers and further reduce data transmission, two types of application-layer messages were designed during the data collection process, and their transmission strategies were proposed.

[0087] During data collection, the satellite terminal constructs application-layer messages and transmits abstract data models via network protocols. Upon receiving the application-layer messages, the satellite extracts the abstract data models and reconstructs the data. Application-layer messages include two types: Out-of-range Value Messages (OVMs) and Aggregated Models Messages (AMMs). To identify these two types of messages, this section uses a single flag bit. A value of 1 indicates OVM, and a value of 0 indicates AMM.

[0088] OVM is used for the timely and accurate uploading of out-of-bounds data. An OVM contains a tag bit and one raw sensing data point. The construction of AMM primarily considers energy efficiency. An AMM contains a tag bit and several abstract data models. Since network protocols have an upper limit on the length of data packets they support, denoted as... Since the length of an abstract data model is Then an AMM can carry An abstract data model. Through AMM, Each value can be sent in a single message, which greatly reduces the amount of data sent, and this process is independent of changes in the environment.

[0089] To ensure timely transmission of environmental anomalies, an OVM (Original Virtual Machine) is uploaded to the satellite immediately upon its creation. The AMM (Automatic Data Model) upload, however, aims to further reduce data transmission volume. Each satellite terminal records an abstract data model of the most recently uploaded regular data and caches it as parameters. When new routine values ​​are collected, they are converted into an abstract data model and compared with this parameter. If they are the same, the data is considered unchanged, and this data is not transmitted. If the satellite does not receive data within the expected time, the data is considered unchanged, and the most recent routine data is used as the currently collected data. If the abstract data model and...

[0090] If they differ, then starting from this data point, construct an AMM from the next k data points and upload it to the satellite. After uploading, update... Since data values ​​collected close together are usually similar, this strategy can further reduce data transmission.

Claims

1. A space-based Internet of Things (IoT) data processing method driven by dual constraints of network and service quality, characterized by: (1) Accuracy-based adaptation mode An accuracy-based adaptation mode ensures that the user-defined error tolerance is met. That is, with accuracy requirements as the core, within the available effective bandwidth Under the constraints, the actual bandwidth usage should be minimized as much as possible. The specific process is as follows: First, fix the error tolerance set by the user. According to the range of data changes Calculate the minimum number of partitions required to divide the data interval. It needs to cover all possible data values: Because it is necessary to distinguish between two types of out-of-bounds states where "data exceeds the normal range"—namely, upper bound out-of-bounds and lower bound out-of-bounds—it is necessary to calculate the minimum number of bits required to support n normal ranges + 2 out-of-bounds states. : ;where ceil is the floor function, ensuring that the number of bits in the encoding can cover all states; Based on the data density of the transmission scenario That is, the number of data units that need to be transmitted per unit time, based on the available effective bandwidth. Calculate the maximum number of bits that the current bandwidth can support for coding. That is, "maximum effective number of bits": floor is a floor function to prevent the actual bandwidth from exceeding the effective upper limit; If the effective bandwidth can support the accuracy, that is... Directly adopt As the final encoding bit length, the data approximation error is strictly controlled at this point. Within, and bandwidth usage To reach the minimum value under the current precision; if the effective bandwidth cannot support the precision. This means that the current bandwidth cannot support the given accuracy and needs to be increased. With bandwidth adaptation: fixed encoding bit width Calculate the maximum number of intervals that can be covered at this time. : ; Calculate the maximum allowed interval width: Adjust the error tolerance to ε_new: Ultimately As a practical accuracy standard, it ensures that within the effective bandwidth capacity, the actual bandwidth usage is within the specified range. ; (2) Or a bandwidth-based adaptation mode The core objective of this mode remains to fully utilize the bandwidth resources set by the user, but it must be based on available and effective bandwidth. Fully configured, not nominal bandwidth The specific process has been adjusted as follows: First, based on data density and available effective bandwidth The number of encoding bits l is fixed to the "effective maximum number of encoding bits" l_max_eff to ensure that the effective bandwidth is fully utilized: At this point, the actual total bit rate ; according to Calculate the maximum number of intervals that can be covered. It contains 2 superboundary states; Combined with the range of data changes By reverse calculation, the maximum allowed interval width can be determined as follows: ;like Unable to be Integer division and rounding up ensure coverage, thus obtaining the minimum error tolerance under the current effective bandwidth: ;Will As the ultimate standard of accuracy.

Images