A data sharing method and system based on metadata

Abstract

This invention relates to the fields of IoT data synchronization, cloud computing, and edge computing technologies, and discloses a data sharing method and system based on metadata. The IoT data synchronization method includes steps such as extracting the sensor's main update cycle through periodic detection, generating sensor semantic embedding vectors through a semantic embedding network, grouping sensors through two-dimensional clustering, generating a scheduling plan by calculating a synchronization time window, and generating a synchronization package by packaging semantic association metadata and reporting it to the cloud. The IoT data synchronization system includes a sensor feature extraction module, a periodic detection module, a semantic embedding module, a semantic association calculation module, a two-dimensional clustering module, a scheduling plan generation module, a synchronization package generation module, and a cloud index establishment module. This invention solves the technical problem of efficient synchronization and semantic association maintenance of heterogeneous IoT data.

Description

Technical Field

[0001] This invention relates to the fields of Internet of Things (IoT) data synchronization, cloud computing, and edge computing technologies; more specifically, it relates to a data sharing method and system based on metadata. Background Technology

[0002] In cloud-based smart park IoT platforms, various types of sensor devices are typically deployed, including temperature sensors, humidity sensors, light sensors, people flow sensors, and energy consumption sensors. Each type of sensor has different sampling frequencies and data update cycles. The system needs to synchronize the data collected by these heterogeneous sensors to the cloud to support comprehensive cloud-based analysis of multi-dimensional data such as park environment, personnel activity, and resource consumption.

[0003] In existing IoT data synchronization methods, data from each sensor is usually reported independently by each device, and data synchronization is achieved through a one-way data flow between the device, gateway, and cloud.

[0004] Existing IoT data synchronization methods suffer from two major drawbacks: First, data from different types of sensors is reported independently by device. When the number of sensors is large, the massive number of fine-grained synchronization requests leads to excessive communication protocol overhead and increased gateway processing pressure. Second, different types of sensors report data independently at different times due to their different sampling periods. The synchronized data lacks cross-type semantic association information. When performing cross-type association analysis in the cloud (such as analyzing the causal relationship between increased pedestrian traffic and increased energy consumption), additional data association mining operations are required. Furthermore, the inconsistent synchronization timing makes time alignment difficult, resulting in low analysis efficiency. This makes the efficient synchronization and semantic association maintenance of heterogeneous IoT data a critical technical problem. Summary of the Invention

[0005] This invention provides a data sharing method and system based on metadata, which solves the technical problems of efficient synchronization and semantic association maintenance of heterogeneous Internet of Things data in related technologies.

[0006] This invention provides a metadata-based data sharing method, comprising:

[0007] Obtain historical data reporting records and metadata definitions of each sensor in the IoT device group, and extract the sensor feature attribute set. The metadata definition includes sensor type, measurement unit, value range and sampling frequency.

[0008] Based on the reported timestamp sequence, the main update cycle of each sensor is extracted using a periodic detection algorithm to generate a sensor periodic feature vector.

[0009] Sensor metadata attributes are input into a semantic embedding network to generate semantic embedding vectors for each sensor.

[0010] Based on a preset set of semantic anchors, the similarity between the semantic embedding vector of each sensor and each anchor is calculated, and a sensor-anchor association matrix is ​​generated. The semantic anchors include environmental state anchors, personnel activity anchors, and resource consumption anchors.

[0011] The sensor periodic feature vector is concatenated with the sensor-anchor point association matrix and input into a two-dimensional clustering algorithm to generate a sensor grouping mapping table. The two-dimensional clustering algorithm performs clustering based on two dimensions: update cycle similarity and semantic anchor point affinity.

[0012] The execution process of the two-dimensional clustering algorithm includes:

[0013] The sensors are divided into candidate groups with adjacent periods based on their periodic characteristics;

[0014] Within each candidate group, a secondary division is performed based on semantic anchor proximity, so that sensors within the same semantic synchronization group have both adjacent update cycles and adjacent semantic attributes.

[0015] Traverse each semantic synchronization group, calculate the group synchronization time window based on the main update cycle of the sensors within the group, and generate a group synchronization scheduling plan.

[0016] When the group synchronization time window arrives, incremental data from each sensor in the group is collected, semantic association metadata is added, and a semantic association synchronization package is generated. The semantic association metadata includes semantic anchor tags and association weights.

[0017] The semantic association synchronization package is sent to the cloud, and the cloud parses the semantic association metadata to establish a cross-type association index structure.

[0018] The similarity is obtained by dividing the dot product of the semantic embedding vector and the semantic anchor vector by the product of the norms of the two vectors; the association weight is obtained by calculating the cosine similarity between the semantic embedding vectors of any two sensors within the group.

[0019] Furthermore, the execution of the periodicity detection algorithm includes:

[0020] Perform a difference operation on the reporting timestamp sequence of each sensor to calculate the interval sequence between adjacent reporting timestamp times;

[0021] Autocorrelation analysis is performed on time interval sequences to identify periodic patterns and extract the main update period value;

[0022] The main update cycle values ​​of each sensor are combined to generate a sensor cycle feature vector.

[0023] Furthermore, the step of inputting sensor metadata attributes into a semantic embedding network to generate semantic embedding vectors for each sensor includes:

[0024] Perform one-hot encoding on the sensor type attribute to generate a type encoding vector;

[0025] Numerical normalization is performed on the measurement unit and value range attributes to generate numerical feature vectors;

[0026] The type encoding vector and the numerical feature vector are concatenated and input into the semantic embedding network, which is a multi-layer fully connected neural network that outputs the semantic embedding vector of each sensor.

[0027] Furthermore, the training process of the semantic embedding network is as follows:

[0028] Obtain a sample set of sensor metadata labeled with semantic categories;

[0029] Sensor pairs with the same semantic category are used as positive samples, and sensor pairs with different semantic categories are used as negative samples.

[0030] The contrastive learning loss function is used for training, which makes semantically adjacent sensors closer together in the embedding space.

[0031] Furthermore, a time window smoothing mechanism is introduced when performing two-dimensional clustering:

[0032] For the similarity calculation of periodic features, the average period within the sliding time window is used to reduce the impact of periodic fluctuations caused by occasional reporting delays on the grouping results.

[0033] Furthermore, the calculation of the group synchronization time window based on the main update cycle of the group's sensors includes:

[0034] For each semantic synchronization group, obtain the set of main update cycles for all sensors within the group;

[0035] Calculate the least common multiple of the main update cycles of the sensors within the group, and use it as the length of the batch synchronization time window for that group;

[0036] Based on the synchronization time window length of each group, a group synchronization scheduling plan is generated.

[0037] Furthermore, a time offset mechanism is introduced when generating the group synchronization scheduling plan:

[0038] Add a random offset to the synchronization trigger time point of each group so that the synchronization operations of different groups are staggered in time.

[0039] Furthermore, the process of generating the semantically associated synchronization packet includes:

[0040] When the synchronization time window of a certain semantic synchronization group is reached, the sensors in the group are traversed to collect incremental data since the last synchronization.

[0041] Extract the semantic anchor labels of each sensor in the group from the sensor-anchor association matrix. The semantic anchor labels are the anchor categories with the highest association degree for each sensor.

[0042] Based on the semantic embedding vectors of sensors within a group, the association weights between sensors within the group are calculated. The association weights represent the semantic association strength between sensor data.

[0043] Incremental data, semantic anchor tags, and associated weight metadata are aggregated and packaged to generate a semantic association synchronization package.

[0044] Furthermore, the cloud-based cross-type association index structure built on semantic association metadata includes:

[0045] The cloud data receiving service parses the synchronization package and extracts incremental data, semantic anchor tags, and associated weight metadata.

[0046] Based on semantic anchor tags, incremental data is stored in the corresponding data partitions according to semantic categories;

[0047] Based on the associated weight metadata, an associated index structure for cross-type sensor data is established. This associated index structure supports the quick retrieval of semantically related sensor data from other categories based on any sensor data.

[0048] This invention provides a metadata-based data sharing system, comprising:

[0049] The sensor feature extraction module is used to obtain the historical data reporting records and metadata definitions of each sensor in the IoT device group, and extract the sensor feature attribute set;

[0050] The period detection module is used to extract the main update period of each sensor based on the reported timestamp sequence and to generate a sensor period feature vector.

[0051] The semantic embedding module is used to input sensor metadata attributes into the semantic embedding network and generate semantic embedding vectors for each sensor.

[0052] The semantic association calculation module is used to calculate the similarity between the semantic embedding vector of each sensor and each anchor point based on a preset set of semantic anchor points, and generate a sensor-anchor point association matrix.

[0053] The two-dimensional clustering module is used to concatenate the sensor periodic feature vector with the sensor-anchor point association matrix and input it into the two-dimensional clustering algorithm to generate a sensor grouping mapping table.

[0054] The scheduling plan generation module is used to traverse each semantic synchronization group, calculate the group synchronization time window based on the main update cycle of the sensors within the group, and generate a group synchronization scheduling plan.

[0055] The synchronization package generation module is used to collect incremental data from each sensor in the group when the group synchronization time window arrives, attach semantic association metadata, and generate a semantic association synchronization package.

[0056] The cloud-based index building module is used to send semantic association synchronization packages to the cloud, and the cloud parses semantic association metadata to build a cross-type association index structure.

[0057] The beneficial effects of this invention are as follows:

[0058] This invention maps the metadata attributes of heterogeneous sensors to a unified semantic vector space and establishes semantic associations between different types of sensors using preset semantic anchors. This enables different types of sensors, such as temperature sensors, people flow sensors, and energy consumption sensors, to be associated and identified based on semantic affinity rather than just device type, thereby improving the accuracy and effectiveness of the association.

[0059] This invention identifies the data update cycle of each sensor through a periodic detection algorithm and performs two-dimensional clustering in conjunction with a semantic anchor point association matrix. This groups heterogeneous sensors with similar update cycles and semantic correlation into the same semantic synchronization group, enabling semantically related sensor data to be synchronized in batches within the same time window. This overcomes the problems of excessive communication protocol overhead and increased gateway processing pressure caused by independent reporting by each sensor, and improves the efficiency of data synchronization.

[0060] This invention carries semantic anchor tags and associated weight metadata in the synchronization package, enabling the cloud to directly establish an associated index structure for cross-type sensor data based on the metadata after receiving the data, without the need for additional data association mining operations. This overcomes the problems of time alignment difficulties caused by inconsistent synchronization timing and low analysis efficiency caused by missing association information. It achieves an efficient data management mechanism of one-time synchronization, multi-dimensional association, and persistent indexing, resulting in improved data synchronization efficiency, improved accuracy of association analysis, and reduced cloud processing burden. Attached Figure Description

[0061] Figure 1 This is a flowchart of the metadata-based data sharing method of the present invention;

[0062] Figure 2 This is a bar chart showing the sensor sampling frequency distribution of the present invention;

[0063] Figure 3 This is a bar chart showing the distribution of sensor semantic classification in this invention;

[0064] Figure 4 This is a bar chart showing the semantic synchronization group time window configuration of the present invention;

[0065] Figure 5 This is a bar chart comparing the data reporting volume of the synchronization group according to the present invention. Detailed Implementation

[0066] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0067] At least one embodiment of the present invention discloses a data sharing method based on metadata, such as Figure 1 As shown, it includes the following steps:

[0068] Step 100: Obtain the historical data reporting records and metadata definitions of each sensor in the IoT device group, and extract the sensor feature attribute set.

[0069] Step 101: Obtain historical data reporting records of each sensor in the device group from the IoT device management platform. The historical data reporting records include a reporting timestamp sequence and a data change amplitude sequence.

[0070] Step 102: Obtain the metadata definition of each sensor from the device registration information. The metadata definition includes sensor type, measurement unit, value range, and sampling frequency.

[0071] Step 103: Associate the reported timestamp sequence, the data change amplitude sequence, and the metadata definition to generate a set of characteristic attributes for each sensor.

[0072] Taking the seven sensors in Office Building 1 of a smart park as an example, the gateway obtains the following recent sensor data records:

[0073] Table 1 Sensor Metadata Definition Table:

[0074]

[0075] Table 2 Example of the reporting timestamp sequence for sensor T001 (temperature):

[0076]

[0077] Through the association processing in step 103, each sensor generates a set of feature attributes containing information such as metadata definition, timestamp sequence, and data change magnitude, providing input data for subsequent periodic detection and semantic embedding.

[0078] Step 200: Based on the reported timestamp sequence, use the periodic detection algorithm to extract the main update period of each sensor and generate the sensor periodic feature vector.

[0079] Step 201: Perform a difference operation on the reporting timestamp sequence of each sensor to calculate the time interval sequence between adjacent reporting timestamps. Further, the specific method of the difference operation is as follows: for the sensor reporting timestamp sequence... ,in For the first A reporting timestamp, Calculate the time interval sequence to represent the total number of timestamps reported by the sensor. That is, the difference between two adjacent timestamps, which gives the set of reporting intervals for each sensor.

[0080] Step 202: Perform autocorrelation analysis on the time interval sequence to identify periodic patterns and extract the main update period value. The input to the autocorrelation analysis is the time interval sequence of each sensor, and the output is the main update period value reported by each sensor. Further, the selection criterion for the main update period value is: from all periodic components obtained from autocorrelation analysis or frequency domain transformation, select the period value corresponding to the periodic component with the largest amplitude as the main update period, because the periodic component with the largest amplitude represents the most significant periodic characteristic of the time interval sequence.

[0081] Step 203: Combine the main update cycle values ​​of each sensor to generate a sensor periodic feature vector. ,in Indicates the first The main update cycle of each sensor This represents the total number of sensors.

[0082] Periodic detection was performed on the seven sensors in steps 100-103. By analyzing the time interval patterns in their historical reported timestamp sequences, the most frequent periodic components were identified using autocorrelation analysis or fast Fourier transform. The main update period results extracted from each sensor are as follows:

[0083] Table 3: Extraction results of the main update cycle for each sensor:

[0084]

[0085] This generates the sensor periodic feature vector. The elements are arranged in order of sensor ID.

[0086] The aforementioned period detection algorithm can employ the Fast Fourier Transform (FFT) method. The input of this method is the time interval sequence of each sensor. By converting the time interval sequence from the time domain to the frequency domain, the period component corresponding to the peak in the spectrum is identified, and the output is the main update period value of each sensor.

[0087] Figure 2 The sampling frequency (main update cycle) distribution of the 7 sensors is shown.

[0088] Step 300: Input the sensor metadata attributes into the semantic embedding network to generate semantic embedding vectors for each sensor.

[0089] Step 301: Perform one-hot encoding on the sensor type attribute to generate a type encoding vector.

[0090] Step 302: Perform numerical normalization on the measurement unit and range attributes to eliminate the influence of dimensions on subsequent calculations and generate numerical feature vectors.

[0091] Step 303: Concatenate the type encoding vector and the numerical feature vector, and input them into the semantic embedding network to generate semantic embedding vectors for each sensor. ,in Represents a d-dimensional real vector space. This refers to the embedding vector dimension. Further, the embedding vector dimension... The value is typically set to a power of 2, such as 32, 64, 128, or 256. The specific value is selected based on the number of sensor types and the required expressive power. For example, when the preset number of semantic anchors is 3, the value can be selected as follows: or .

[0092] After one-hot encoding and normalization of the metadata attributes (sensor type, measurement unit, value range) of the seven sensors, they are input into the trained semantic embedding network (in this example, dimensionality is used). This generates embedding vectors for each sensor in the semantic space. Examples of the first four dimensions of the generated embedding vectors for each sensor are shown below:

[0093] Table 4 Examples of sensor semantic embedding vectors (first 4 dimensions):

[0094]

[0095] The table shows the values ​​of the first four dimensions of the semantic embedding vector for each sensor. The distribution of vector values ​​indicates that sensors of the same semantic category (such as T001, H001, and L001, all belonging to the environmental monitoring category) have embedding vectors positioned close together in the semantic space, while sensors of different categories (such as environmental monitoring and human activity) have vectors that are spaced further apart.

[0096] The semantic embedding network is a multi-layer fully connected neural network. The input layer of this network receives the concatenated feature vector, which undergoes a nonlinear transformation through multiple hidden layers. The output layer outputs the embedding vector of each sensor in the semantic space. Let the concatenated feature vector of the sensor metadata be... The computation process of the semantic embedding network is as follows: , , ,in For activation function, and The first The weight matrix and bias vector of the layer, The total number of network layers. Indicates the first The hidden states of the layers. Furthermore, the total number of layers in the network. The value of is an integer between 2 and 6. This indicates the number of hidden layers. There must be at least one hidden layer; the input layer and the output layer each count as one layer. For example... This indicates a total of three layers: an input layer, one hidden layer, and an output layer. Furthermore, the activation function... The rectified linear unit (ReLU) is used, which is defined as follows: It features high computational efficiency and can effectively alleviate the gradient vanishing problem; the output layer does not apply an activation function and directly outputs the embedding vector.

[0097] The aforementioned semantic embedding network is trained using a contrastive learning paradigm, on a set of labeled sensor metadata samples. For sensor pairs with the same semantic category... As positive samples, sensor pairs with different semantic categories are used as negative samples, and the InfoNCE contrastive learning loss function is used for training:

[0098]

[0099] in, Represents the natural logarithm function. The cosine similarity between two vectors is represented by the following expression: For temperature parameters, The total number of samples in the batch. Represents an exponential function. This is the index variable for the samples in the batch, with a value range of 1 to... During network training, a mini-batch gradient descent optimization strategy is employed, ensuring that semantically similar sensors are positioned close together in the embedding space, while semantically different sensors are positioned further apart. Furthermore, the optimization objective is to minimize the contrastive loss function. The goal is to achieve this by maximizing the similarity of semantically related sensor pairs over time (i.e., within a training period). The goal is to maximize the similarity between sensors with different semantics and minimize the similarity, thereby ensuring the discriminativeness of the embedded vectors in the semantic space; the constraint is... Furthermore, the temperature parameter The value of is a positive number between 0 and 1. This parameter controls the smoothness of the similarity distribution in contrastive learning. A smaller value results in a smoother distribution. A higher value makes the model more sensitive to differences in similarity; a larger value indicates a higher sensitivity. The value makes the distribution smoother; for example, a value of 0.07 can be used.

[0100] Step 400: Based on the preset set of semantic anchors, calculate the cosine similarity between the semantic embedding vector of each sensor and each anchor, and generate the sensor-anchor association matrix.

[0101] Step 401: Obtain the preset set of semantic anchor points The semantic anchors are predefined reference vectors in the semantic space, including environmental state anchors, personnel activity anchors, and resource consumption anchors. This represents the number of anchor points.

[0102] Step 402: For the semantic embedding vector of each sensor Calculate its relationship with each semantic anchor point Cosine similarity:

[0103]

[0104] in, Indicates the first The sensor and the first Similarity of anchor points This represents the Euclidean norm of a vector.

[0105] Step 403: Organize the similarity values ​​between each sensor and each anchor point into a sensor-anchor point association matrix. .

[0106] This example presupposes three semantic anchors: environment state anchor ( ), personnel activity anchor points ( Resource consumption anchor ( Based on the sensor embedding vectors generated in step 303, the cosine similarity between each sensor and each anchor point is calculated, resulting in the sensor-anchor point association matrix as follows:

[0107] Table 5 Sensor-Anchor Point Correlation Matrix:

[0108]

[0109] The values ​​in the matrix represent the similarity between each sensor and the semantic anchor point. The matrix shows that environmental sensors (T001, H001, L001) have the highest similarity to environmental state anchor points. The human activity sensors (P001, D001) showed the highest similarity to human activity anchor points. Resource consumption sensors (E001, W001) showed the highest similarity to resource consumption anchor points. This demonstrates the rationality of the anchor point definition.

[0110] It should be noted that the semantic anchor points mentioned above are set as follows: environmental state anchor points correspond to the semantic centers of environmental monitoring sensors such as temperature, humidity, and light intensity; personnel activity anchor points correspond to the semantic centers of personnel activity monitoring sensors such as pedestrian flow, access control, and attendance; and resource consumption anchor points correspond to the semantic centers of resource consumption monitoring sensors such as energy consumption, water consumption, and gas consumption. For each semantic category, its corresponding semantic anchor point vector... This is obtained by taking the arithmetic mean of the semantic embedding vectors of all sensors in this category, i.e. ,in Indicates belonging to the first A set of sensors with semantic categories. The number of sensors in this set. For traversing the set Index variables for each sensor in the system.

[0111] Step 500: Concatenate the sensor periodic feature vector with the sensor-anchor point association matrix and input it into the two-dimensional clustering algorithm to generate a sensor grouping mapping table.

[0112] Step 501: Process the sensor periodic feature vector Normalization is performed to eliminate the influence of dimensions on subsequent clustering operations. Then, the normalized periodic feature vectors are compared with the sensor-anchor correlation matrix. Concatenate columns to generate a sensor integrated feature matrix. .

[0113] Step 502: Perform two-dimensional clustering on the comprehensive feature matrix to divide the heterogeneous sensors into multiple semantic synchronization groups. The input of the two-dimensional clustering is the sensor comprehensive feature matrix, and the output is a sensor grouping mapping table. The two-dimensional clustering algorithm includes the following sub-steps:

[0114] Step 5021: Extract the periodic feature column from the sensor integrated feature matrix and calculate the periodic difference value between each sensor. For the sensor and sensors The period difference value is .

[0115] Step 5022: Based on the period difference value, the sensors are clustered in the first dimension. Using a hierarchical clustering method based on the principle of period similarity, sensors with period difference values ​​less than a preset period threshold are grouped into the same candidate group, generating several period candidate groups. Further, the preset period threshold is determined as follows: The difference value statistics (such as the maximum difference value or standard deviation) in the main update period set of all sensors are calculated. Then, according to the application scenario's requirements for grouping granularity, a certain proportion (e.g., 20%–40%) of this difference value statistics is set as the period threshold to achieve different clustering granularities. For sensors… and To group both into the same candidate group, the following constraints must be met: ,in A preset periodic threshold is used to ensure that the difference in update periods among sensors within the same group is within an acceptable range.

[0116] Step 5023: For each periodic candidate group, extract the semantic anchor point association vector of each sensor within the group in the sensor-anchor point association matrix. Further, the semantic anchor point association vector refers to the sensor's association vector within the sensor-anchor point association matrix. The corresponding row vector in the sensor Its semantic anchor association vector is This vector contains the information about the sensor and all... The similarity value of each semantic anchor point.

[0117] Step 5024: Within each candidate group of a period, calculate the semantic distance between sensors within the group. For sensors within the group... and sensors The semantic distance is the Euclidean distance between the semantic anchor vectors of the two entities. ,in and Sensors and sensors The semantic anchor associated vector.

[0118] Step 5025: Perform second-dimensional clustering within each periodic candidate group. Use the K-means clustering method to further divide the sensors within the group into several semantic synchronization groups based on semantic distance. This ensures that sensors within the same semantic synchronization group have both similar update cycles and similar semantic attributes. Further, the K value in the K-means clustering is determined as follows: Based on the number of sensors within the periodic candidate group and the preset number of semantic anchors, the K value is set to the number of semantic anchors or an integer value close to it. For example, when the number of semantic anchors is 3, the K value can be set to an integer between 2 and 4. The specific value is adjusted according to the semantic distribution characteristics of the sensors within the group.

[0119] Step 503: Generate a sensor grouping mapping table, which records the semantic synchronization group identifier to which each sensor belongs.

[0120] Based on the periodic feature vector generated in step 200 Based on the sensor-anchor association matrix generated in step 400, perform two-dimensional clustering (the first dimension classifies based on periodic similarity, and the second dimension classifies based on semantic distance). Set a periodic threshold. The grouping results are as follows:

[0121] Table 6 Sensor Grouping Mapping Table:

[0122]

[0123] The grouping results show that the system aggregates 7 sensors into 5 semantic synchronization groups based on similarity of cycles and semantic relevance: environmental monitoring fast synchronization group (G1, cycle 30 seconds), personnel activity synchronization group (G2, G4, cycle 30-60 seconds), resource consumption synchronization group (G5, cycle 120 seconds), etc., realizing semantic association grouping of heterogeneous sensors.

[0124] The aforementioned two-dimensional clustering algorithm uses the periodic mean within a sliding time window instead of a single detection value when performing periodic feature similarity calculation, thereby reducing the impact of periodic fluctuations caused by occasional reporting delays on the grouping results.

[0125] Figure 3 The distribution of the seven sensors across three semantic categories is shown.

[0126] Step 600: Traverse each semantic synchronization group, calculate the group synchronization time window based on the main update cycle of the sensors within the group, and generate a group synchronization scheduling plan.

[0127] Step 601: For each semantic synchronization group, obtain the set of main update cycles for all sensors within the group.

[0128] Step 602: Calculate the least common multiple of the main update cycles of the sensors within the group, and use this as the batch synchronization time window length for the group. Further, the batch synchronization time window length... The calculation satisfies the following constraints: ,in The function represents the least common multiple. For the first Semantic synchronization groups, This represents the number of sensors within the group. For the first The main update cycle of each sensor; at the same time, the time window length should meet the following requirements. ,in This represents the maximum value function, ensuring that the time window can include at least one complete update cycle for all sensors in the group.

[0129] Step 603: Based on the synchronization time window length of each group, generate a group synchronization scheduling plan, which includes the synchronization trigger time point sequence of each group.

[0130] For the five semantic synchronization groups obtained in step 503, calculate the batch synchronization time window length for each group based on the main update cycle of the sensors within the group (using the least common multiple algorithm):

[0131] Table 7 shows the calculation results for the synchronization time windows:

[0132]

[0133] Based on the synchronization time window length of each group and the preset synchronization trigger start point, a group synchronization scheduling plan is generated. To avoid network congestion, the system adds a random offset (ranging from 0 to 10% of the window length) to the synchronization trigger time point of each group. In this example, the gateway's base time is assumed to be... The actual synchronization trigger time sequence for each group is as follows:

[0134] Table 8 Synchronization Scheduling Plan:

[0135]

[0136] The scheduling plan shows that the synchronization operations of each semantic synchronization group are evenly distributed in time, avoiding the concentration of synchronization time points and effectively reducing the risk of network congestion.

[0137] In this embodiment, to avoid network congestion caused by the excessive concentration of synchronization times of multiple semantic synchronization groups, a random offset is added to the synchronization trigger time of each group when generating the group synchronization scheduling plan, so that the synchronization operations of different groups are staggered in time. Furthermore, the value range of the random offset is from 0 to a certain proportion of the synchronization time window length of the group, for example, from 0 to 10% of the window length, thereby ensuring that the offset will not exceed one synchronization cycle and preventing excessively dispersed synchronization times from causing excessively long system response delays.

[0138] Figure 4 Display the time window lengths of the 5 semantic synchronization groups.

[0139] Step 700: When the group synchronization time window arrives, collect incremental data from each sensor in the group, attach semantic association metadata, and generate a semantic association synchronization package.

[0140] Step 701: When the synchronization time window of a certain semantic synchronization group arrives, iterate through each sensor in the group and collect incremental data since the last synchronization.

[0141] Step 702: Extract the semantic anchor tags for each sensor within the group from the sensor-anchor association matrix. The semantic anchor tags are the anchor categories with the highest association degree for each sensor. Further, the specific extraction method for the semantic anchor tags is as follows: for each sensor... In the sensor-anchor point correlation matrix Find the row vector corresponding to the sensor The maximum value in the range, and its corresponding column index. This refers to the semantic anchor label of the sensor, where Indicates taking such that Index that reaches the maximum value The extraction process satisfies the following constraints: That is, the semantic anchor label can only be determined when the sensor has a non-negative semantic similarity with at least one anchor.

[0142] Step 703: Based on the semantic embedding vectors of the sensors within the group, calculate the association weights between the sensors within the group, whereby the association weights represent the semantic association strength between the sensor data.

[0143] Step 704: Aggregate and package the incremental data, semantic anchor tags, and associated weight metadata to generate a semantic association synchronization package.

[0144] When semantic synchronization group G1 is in relative time When the synchronization time window is reached, the gateway performs steps 701-704 on each sensor (T001, H001) within G1. First, incremental data is collected since the last synchronization. Then, the semantic anchor label for each sensor is extracted from the sensor-anchor association matrix generated in step 400. According to the data in the table, the semantic anchor label for T001 is... (Environmental state anchor point), the semantic anchor tag for H001 is... (Environmental state anchor point).

[0145] Subsequently, the association weight between sensors T001 and H001 within G1 is calculated: (Exceeding the preset threshold of 0.5), therefore this associated weight is recorded in the synchronization package.

[0146] Table 9. Example of G1 semantic association synchronization package content:

[0147]

[0148] It should be noted that the above-mentioned correlation weights are calculated as follows: for any two sensors within a group... and Its correlation weight The cosine similarity value of the semantic embedding vectors of the two is calculated using the following formula: The association weight matrix is ​​stored in the synchronization packet in a sparse matrix format, retaining only sensor pairs with weight values ​​exceeding a preset threshold. Furthermore, the preset threshold is used to filter weight values; this threshold is set between 0 and 1, and the specific value is configured according to the application scenario's requirements for association strength. For example, setting it to 0.5 means only retaining sensor pairs with a similarity greater than 0.5. Simultaneously, during synchronization packet generation, constraints must be met: only those satisfying the following conditions... (in Sensor pairs (with a preset threshold) Only then are they recorded in the sparse matrix. This filtering is performed within each synchronization time window to ensure the consistency of time synchronization.

[0149] Figure 5 Displays the data reporting volume of each synchronization group within a complete synchronization cycle.

[0150] Step 800: Send the semantic association synchronization package to the cloud. The cloud parses the semantic association metadata to establish a cross-type association index structure and outputs the synchronization completion status.

[0151] Step 801: Send the semantic association synchronization package to the cloud data receiving service through the network communication interface.

[0152] Step 802: The cloud data receiving service parses the synchronization package and extracts incremental data, semantic anchor tags, and associated weight metadata.

[0153] Step 803: Based on semantic anchor tags, store incremental data into the corresponding data partitions according to semantic categories.

[0154] Step 804: Based on the associated weight metadata, establish an associated index structure for cross-type sensor data. The associated index structure supports the quick retrieval of other types of sensor data that are semantically related to any sensor data.

[0155] Step 805: Return the synchronization completion status to the data synchronization processor.

[0156] The cloud data receiving service processes synchronization packets from each semantic synchronization group sequentially. For the synchronization packet of group G1 (see the example table in step 704 for details), the cloud performs the following operations: First, it extracts the semantic anchor tags (environmental state anchors) from the synchronization packet and stores the temperature and humidity data in the environmental data partition; then, it establishes an association index between T001 and H001 based on the weight (T001, H001): 0.92 in the association weight metadata. This allows subsequent cloud analysis to quickly query "given temperature data, retrieve related humidity data".

[0157] After the synchronization packages from multiple semantic synchronization groups arrive at the cloud, the cloud can establish various forms of association indexes based on the semantic anchor tags and association weight metadata provided by each synchronization package. Among these, the index based on association weight metadata comes from the synchronization package in step 704: the association weights provided by group G1 are used to associate temperature and humidity, and the association weights provided by group G5 are used to associate energy consumption and water consumption. Furthermore, the cloud can also perform dimensional-level data aggregation based on semantic anchor tags: temperature, humidity, and light data from the same environmental state anchor can be aggregated into an environmental partition; people flow and access control data from the same personnel activity anchor can be aggregated into a personnel partition; and energy consumption and water consumption data from the same resource consumption anchor can be aggregated into a resource partition.

[0158] Table 10 Summary of Related Indexes Processed in Cloud Synchronization:

[0159]

[0160] After cloud processing is complete, a synchronization completion status is returned to the gateway. Upon receiving confirmation, the gateway continues to wait for the next synchronization time window. Throughout the process, the metadata-based data synchronization method achieves an efficient data management mechanism of one-time synchronization, multi-dimensional association, and persistent indexing through the aggregation of group-level association weight indexes and dimension-level semantic anchor tags.

[0161] This implementation uses a semantic embedding network to map the metadata attributes of heterogeneous sensors to a unified semantic vector space, and establishes semantic associations between different types of sensors through preset semantic anchors, enabling different types of sensors such as temperature sensors, people flow sensors, and energy consumption sensors to be associated and identified based on semantic affinity rather than just device type.

[0162] This implementation identifies the data update cycle of each sensor through a periodic detection algorithm and performs two-dimensional clustering in conjunction with a semantic anchor point association matrix. This groups heterogeneous sensors with similar update cycles and semantic correlation into the same semantic synchronization group, enabling semantically related sensor data to be synchronized in batches within the same time window. This overcomes the problem of excessive protocol overhead caused by independent reporting by each sensor.

[0163] This implementation carries semantic anchor tags and associated weight metadata in the synchronization package, enabling the cloud to directly establish an associated index structure for cross-type sensor data based on the metadata after receiving the data, without the need to perform additional data association mining operations. This overcomes the difficulties in time alignment caused by inconsistent synchronization timing and the low analysis efficiency caused by missing association information, thus solving the dual technical challenges of efficient synchronization and semantic association of heterogeneous IoT data.

[0164] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A data sharing method based on metadata, characterized in that, Includes the following steps: Obtain historical data reporting records and metadata definitions of each sensor in the IoT device group, and extract the sensor feature attribute set. The metadata definition includes sensor type, measurement unit, value range and sampling frequency. Based on the reported timestamp sequence, the main update cycle of each sensor is extracted using a periodic detection algorithm to generate a sensor periodic feature vector. Sensor metadata attributes are input into a semantic embedding network to generate semantic embedding vectors for each sensor. Based on a preset set of semantic anchors, the similarity between the semantic embedding vector of each sensor and each anchor is calculated, and a sensor-anchor association matrix is ​​generated. The semantic anchors include environmental state anchors, personnel activity anchors, and resource consumption anchors. The sensor periodic feature vector is concatenated with the sensor-anchor point association matrix and input into a two-dimensional clustering algorithm to generate a sensor grouping mapping table. The two-dimensional clustering algorithm performs clustering based on two dimensions: update cycle similarity and semantic anchor point affinity. The execution process of the two-dimensional clustering algorithm includes: The sensors are divided into candidate groups with adjacent periods based on their periodic characteristics; Within each candidate group, a secondary division is performed based on semantic anchor proximity, so that sensors within the same semantic synchronization group have both adjacent update cycles and adjacent semantic attributes. Traverse each semantic synchronization group, calculate the group synchronization time window based on the main update cycle of the sensors within the group, and generate a group synchronization scheduling plan. When the group synchronization time window arrives, incremental data from each sensor in the group is collected, semantic association metadata is added, and a semantic association synchronization package is generated. The semantic association metadata includes semantic anchor tags and association weights. The semantic association synchronization package is sent to the cloud, and the cloud parses the semantic association metadata to establish a cross-type association index structure. The similarity is obtained by dividing the dot product of the semantic embedding vector and the semantic anchor vector by the product of the norms of the two vectors; the association weight is obtained by calculating the cosine similarity between the semantic embedding vectors of any two sensors within the group.

2. The data sharing method based on metadata according to claim 1, characterized in that, The execution of the periodicity detection algorithm includes: Perform a difference operation on the reporting timestamp sequence of each sensor to calculate the interval sequence between adjacent reporting timestamp times; Autocorrelation analysis is performed on time interval sequences to identify periodic patterns and extract the main update period value; The main update cycle values ​​of each sensor are combined to generate a sensor cycle feature vector.

3. The data sharing method based on metadata according to claim 1, characterized in that, The step of inputting sensor metadata attributes into a semantic embedding network to generate semantic embedding vectors for each sensor includes: Perform one-hot encoding on the sensor type attribute to generate a type encoding vector; Numerical normalization is performed on the measurement unit and value range attributes to generate numerical feature vectors; The type encoding vector and the numerical feature vector are concatenated and input into the semantic embedding network, which is a multi-layer fully connected neural network that outputs the semantic embedding vector of each sensor.

4. The data sharing method based on metadata according to claim 3, characterized in that, The training process of the semantic embedding network is as follows: Obtain a sample set of sensor metadata labeled with semantic categories; Sensor pairs with the same semantic category are used as positive samples, and sensor pairs with different semantic categories are used as negative samples. The contrastive learning loss function is used for training, which makes semantically adjacent sensors closer together in the embedding space.

5. The data sharing method based on metadata according to claim 1, characterized in that, Introduce a time window smoothing mechanism when performing two-dimensional clustering: For the similarity calculation of periodic features, the average period within the sliding time window is used to reduce the impact of periodic fluctuations caused by occasional reporting delays on the grouping results.

6. The data sharing method based on metadata according to claim 1, characterized in that, The calculation of the group synchronization time window based on the main update cycle of in-group sensors includes: For each semantic synchronization group, obtain the set of main update cycles for all sensors within the group; Calculate the least common multiple of the main update cycles of the sensors within the group, and use it as the length of the batch synchronization time window for that group; Based on the synchronization time window length of each group, a group synchronization scheduling plan is generated.

7. The metadata-based data sharing method according to claim 6, characterized in that, Introduce a time offset mechanism when generating group synchronization scheduling plans: Add a random offset to the synchronization trigger time point of each group so that the synchronization operations of different groups are staggered in time.

8. The data sharing method based on metadata according to claim 1, characterized in that, The process of generating semantically associated synchronization packets includes: When the synchronization time window of a certain semantic synchronization group is reached, the sensors in the group are traversed to collect incremental data since the last synchronization. Extract the semantic anchor labels of each sensor in the group from the sensor-anchor association matrix. The semantic anchor labels are the anchor categories with the highest association degree for each sensor. Based on the semantic embedding vectors of sensors within a group, the association weights between sensors within the group are calculated. The association weights represent the semantic association strength between sensor data. Incremental data, semantic anchor tags, and associated weight metadata are aggregated and packaged to generate a semantic association synchronization package.

9. The data sharing method based on metadata according to claim 1, characterized in that, The cloud-based cross-type association index structure built on semantic association metadata includes: The cloud data receiving service parses the synchronization package and extracts incremental data, semantic anchor tags, and associated weight metadata. Based on semantic anchor tags, incremental data is stored in the corresponding data partitions according to semantic categories; Based on the associated weight metadata, an associated index structure for cross-type sensor data is established. This associated index structure supports the quick retrieval of semantically related sensor data from other categories based on any sensor data.

10. A metadata-based data sharing system, used to execute the metadata-based data sharing method according to any one of claims 1 to 9, characterized in that, include: The sensor feature extraction module is used to obtain the historical data reporting records and metadata definitions of each sensor in the IoT device group, and extract the sensor feature attribute set; The period detection module is used to extract the main update period of each sensor based on the reported timestamp sequence and to generate a sensor period feature vector. The semantic embedding module is used to input sensor metadata attributes into the semantic embedding network and generate semantic embedding vectors for each sensor. The semantic association calculation module is used to calculate the similarity between the semantic embedding vector of each sensor and each anchor point based on a preset set of semantic anchor points, and generate a sensor-anchor point association matrix. The two-dimensional clustering module is used to concatenate the sensor periodic feature vector with the sensor-anchor point association matrix and input it into the two-dimensional clustering algorithm to generate a sensor grouping mapping table. The scheduling plan generation module is used to traverse each semantic synchronization group, calculate the group synchronization time window based on the main update cycle of the sensors within the group, and generate a group synchronization scheduling plan. The synchronization package generation module is used to collect incremental data from each sensor in the group when the group synchronization time window arrives, attach semantic association metadata, and generate a semantic association synchronization package. The cloud-based index building module is used to send semantic association synchronization packages to the cloud, and the cloud parses semantic association metadata to build a cross-type association index structure.

Images