Intelligent data acquisition and edge computing method and system

Abstract

The application provides a kind of intelligent data acquisition and edge computing method and system, it is related to edge computing and data processing technical field, including by predicting edge node load fluctuation law, parallel splitting and merging batch processing mode of dynamic switching data flow.Extract and verify the causal dependence of structured data to determine effective data, and then group computing statistical characteristics.Construct information flow network based on aggregation results, identify receiving and sending nodes and generate corresponding frequency reduction or enhancement strategies to adjust the sampling parameters of data sources.The application realizes the dynamic optimization of load adaptive processing and data acquisition strategy, improves the utilization efficiency of edge computing resources and data value density.

Description

Technical Field

[0001] This invention relates to the field of edge computing and data processing technology, and in particular to an intelligent data acquisition and edge computing method and system. Background Technology

[0002] In the fields of the Internet of Things (IoT) and the Industrial Internet, intelligent data acquisition and edge computing are key technologies for achieving real-time monitoring and decision-making. Current conventional practices typically involve deploying edge computing nodes to directly receive continuous data streams from various sensors or device data sources. These nodes are responsible for preliminary parsing, filtering, and aggregation of the raw data to reduce the amount of data transmitted to the cloud and lower latency. The data processing flow is often linear, meaning it is processed serially according to the order in which the data arrives, or it uses a fixed batch processing window to accumulate data and then perform unified calculations.

[0003] However, it has significant limitations when processing dynamically changing real-time data streams. The computational load of edge computing nodes fluctuates significantly with changes in the number of connected data sources and the data generation rate. Fixed or simple adaptive processing strategies are difficult to effectively handle such fluctuations. During peak load periods, serial processing can easily lead to data backlog and increased processing latency, potentially resulting in the loss of critical information or delayed decision-making. Conversely, during off-peak periods, fixed batch processing strategies may create unnecessary processing delays while waiting for sufficient data, failing to fully utilize the idle computing resources of nodes and leading to uneven resource utilization.

[0004] Furthermore, existing methods have shortcomings in data validity verification and cross-data source correlation analysis. They typically filter data based on preset rules or single-point statistics, lacking the ability to uncover deep logical relationships between fields within the data, especially causal dependencies. This can lead to invalid or erroneous data being included in subsequent analyses, affecting the accuracy of aggregation results. Simultaneously, for systems composed of multiple related data sources, existing methods often view the characteristics of each data source in isolation, lacking quantitative analysis of information flow and the strength of mutual influence between data sources. This makes it impossible to precisely control the data collection behavior of data sources based on their actual roles in the information network, hindering the optimization of the overall data flow quality and efficiency at the system level. Summary of the Invention

[0005] The embodiments of the present invention provide intelligent data acquisition and edge computing methods and systems, which can solve the problems in the prior art.

[0006] A first aspect of the present invention provides an intelligent data acquisition and edge computing method, comprising:

[0007] Raw data streams are obtained from multiple data sources, and the raw data streams carry data source identifiers;

[0008] The historical load sequence of edge computing nodes is decomposed in the frequency domain to extract periodic features. The load fluctuation pattern is predicted based on the periodic features. During the load increase phase, the original data stream is split into sub-streams for parallel processing. During the load decrease phase, the backlog data stream is merged and batch processed. The original data stream is parsed to obtain structured data.

[0009] Extract the causal dependencies between fields in the structured data, identify the field combinations with causal dependencies, verify the global symbolic consistency of the field combinations through a multi-scale sliding window, and retain the structured data that passes the consistency verification to determine the valid data;

[0010] Calculate statistical features for valid data grouped by data source identifier, and generate aggregated results containing statistical features and data source identifier;

[0011] Based on the aggregation results, the intensity of directional information transmission between data sources is quantified, an information flow network is constructed, the receiving nodes and sending nodes of information in the information flow network are identified, a frequency reduction strategy is generated for the receiving nodes and an enhancement strategy is generated for the sending nodes, and the frequency reduction strategy and enhancement strategy are distributed to the edge computing nodes.

[0012] The sampling parameters of the corresponding data source are adjusted according to the frequency reduction strategy and the enhancement strategy.

[0013] In one optional embodiment, the historical load sequence of the edge computing node is decomposed in the frequency domain to extract periodic features. Based on these periodic features, the load fluctuation pattern is predicted. During the load increase phase, the original data stream is split into sub-streams for parallel processing. During the load decrease phase, the backlogged data stream is merged and batch-processed. The original data stream is then parsed to obtain structured data, including:

[0014] Perform frequency domain transformation on the historical load sequence, extract the frequency component with the largest amplitude as the load cycle feature, and record the cycle length and phase information corresponding to the frequency component.

[0015] Project the current load value onto the phase space of the load cycle characteristics, calculate the future load change slope corresponding to the current phase position, determine the load rising phase when the future load change slope is greater than zero, and determine the load falling phase when the future load change slope is less than zero and the backlog queue length exceeds the preset capacity.

[0016] During the load increase phase, the data dependencies between fields in the original data stream are analyzed. Fields that are directly dependent are grouped into the same field group, and field groups without dependencies are split into independent sub-streams. Each independent sub-stream is assigned a parsing thread for parallel processing, and each parsing thread outputs a structured fragment carrying the field group identifier.

[0017] During the load reduction phase, the data streams in the backlog queue are grouped according to the data source identifier, the data streams from the same data source are merged into batch processing units, and batch parsing is performed on the batch processing units to obtain batch structured data.

[0018] Based on the field group identifier, each structured fragment is concatenated in order according to the original field dependencies. The concatenated structured fragments are then merged with the batch structured data to generate complete structured data.

[0019] In one alternative embodiment, splitting a group of fields without dependencies into independent substreams includes:

[0020] Generate a unique sub-stream identifier for each independent sub-stream;

[0021] Extract the global sequence number of each data record in the original data stream, combine the global sequence number with the substream identifier to generate a composite identifier, and append the composite identifier to the corresponding data record in the independent substream;

[0022] Record the field position index of the field group corresponding to each independent sub-stream in the original data stream, and store the field position index with the sub-stream identifier;

[0023] The independent sub-streams carrying composite identifiers are assigned to the parsing thread for parallel processing;

[0024] The structured fragments output by each parsing thread retain composite identifiers and field position indexes.

[0025] In one optional embodiment, causal dependencies between fields in the structured data are extracted, combinations of fields with causal dependencies are identified, and the global symbolic consistency of the field combinations is verified using a multi-scale sliding window. Structured data that passes the consistency verification is retained to determine valid data, including:

[0026] Extract time series data from each field of structured data;

[0027] A time offset scanning matrix is ​​constructed for the time series data of each field. The time position of the second field relative to the first field is gradually adjusted within a preset time offset range. The change in conditional entropy between the first field and the second field is calculated at each time offset position. The time offset corresponding to the minimum value of the change in conditional entropy is recorded to determine the optimal time delay. When the optimal time delay is positive, the first field is determined to be the cause field and the second field is determined to be the effect field. The cause field, the effect field and the optimal time delay are recorded to form a causal dependency triple.

[0028] Aggregate causal dependency triples that have a common cause field or a common effect field into a field combination;

[0029] Perform time alignment verification on each field combination, generate a delay compensation sequence based on the optimal delay in the causal dependency triple, calculate the global symbol consistency index through a multi-scale sliding window, and mark the field combination as consistent when the global symbol consistency index exceeds the preset consistency threshold;

[0030] Extract the structured data corresponding to the field combinations that pass consistency to determine valid data.

[0031] In an optional embodiment, time-series alignment verification is performed on each field combination. A delay compensation sequence is generated based on the optimal delay in the causal dependency triples. A global symbolic consistency index is calculated using a multi-scale sliding window. When the global symbolic consistency index exceeds a preset consistency threshold, the field combination is marked as consistent, including:

[0032] Based on the optimal time delay in the causal dependency triplet, the time series of the effect-end field is forward-shifted to generate a time delay compensation sequence, and the time series of the cause-end field is used as the reference baseline sequence.

[0033] Multiple sliding windows of different lengths are set, and each sliding window is synchronously slid over the reference sequence and the time delay compensation sequence. The rate of change of the values ​​of the reference sequence and the time delay compensation sequence within the sliding window is calculated.

[0034] For each sliding window, count the number of times the sign of the rate of change of values ​​is consistent within the sliding window, calculate the proportion of the number of times the sign of the rate of change of values ​​is consistent to the length of the sliding window, and determine the local sign matching degree of the sliding window.

[0035] The local symbol matching degree of all sliding windows is weighted and summed according to the weight coefficient corresponding to the sliding window length to obtain the global symbol consistency index of the field combination.

[0036] When the global symbol consistency index exceeds the preset consistency threshold, the field combination is marked as consistent.

[0037] In one optional embodiment, the intensity of directional information transmission between data sources is quantified based on the aggregation results, an information flow network is constructed, and the receiving and sending nodes of information in the information flow network are identified, including:

[0038] Time series alignment is performed on the statistical features of each data source in the aggregation results to extract the time evolution trajectory of the statistical features of each data source;

[0039] Cross-correlation analysis is performed on the time evolution trajectories of statistical characteristics of each data source. The maximum cross-correlation number of the time evolution trajectory of the first data source relative to the time evolution trajectory of the second data source and its corresponding time lag are calculated. When the maximum cross-correlation number exceeds the preset correlation threshold and the time lag is positive, the first data source is determined to be the information sending source and the second data source is determined to be the information receiving source. The maximum cross-correlation number is used as the directional information transmission strength of the data source pair.

[0040] An information flow network is constructed with data sources as nodes, and directed transmission edges are established in the information flow network for each information sending source and information receiving source pair, pointing from the information sending source to the information receiving source.

[0041] Assign edge weight attributes to each directed transmission edge in the information flow network, take the directional information transmission intensity of the data source pair corresponding to the directed transmission edge as the edge weight value of the directed transmission edge, and mark the time lag corresponding to the data source pair for each directed transmission edge in the information flow network as the transmission delay mark.

[0042] Calculate the in-degree and out-degree of each node in the information flow network, mark the nodes with an in-degree greater than their out-degree as information receiving nodes, and mark the nodes with an out-degree greater than their in-degree as information sending nodes.

[0043] In one optional embodiment, generating a frequency reduction strategy for the receiving node and an enhancement strategy for the sending node, and then distributing the frequency reduction strategy and the enhancement strategy to the edge computing node includes:

[0044] Extract the incoming directed transmission edges of each receiving node, calculate the product of the edge weight value and the transmission delay mark to obtain the delay weighting strength, accumulate all delay weighting strengths to obtain the cumulative incoming information, map the cumulative incoming information to the frequency reduction amplitude coefficient, adjust the sampling period of the corresponding data source of the receiving node according to the frequency reduction amplitude coefficient, and generate the frequency reduction strategy.

[0045] Extract the outgoing directed transmission edges of each sending node, filter the edges whose weight values ​​exceed the preset strength threshold to form a key transmission edge set, count the number of edges in the key transmission edge set to obtain the impact coverage metric, extract the minimum value of the transmission delay marker in the key transmission edge set to obtain the primary response delay, multiply the impact coverage metric and the reciprocal of the primary response delay to obtain the enhancement score, map the enhancement score to the enhancement amplitude coefficient, and adjust the sampling frequency of the data source corresponding to the sending node according to the enhancement amplitude coefficient to generate the enhancement strategy;

[0046] The frequency reduction strategy and the enhancement strategy are packaged into a strategy instruction package according to the data source identifier and sent to the edge computing node.

[0047] A second aspect of the present invention provides an intelligent data acquisition and edge computing system, comprising:

[0048] The data acquisition unit is used to acquire raw data streams from multiple data sources, wherein the raw data streams carry data source identifiers;

[0049] The load optimization unit is used to extract periodic features by frequency domain decomposition of the historical load sequence of edge computing nodes, predict the load fluctuation pattern based on the periodic features, split the original data stream into sub-streams for parallel processing during the load increase phase, merge the backlog data streams for batch processing during the load decrease phase, and parse the original data stream to obtain structured data.

[0050] The data validation unit is used to extract the causal dependencies between fields in structured data, identify field combinations with causal dependencies, verify the global symbolic consistency of field combinations through a multi-scale sliding window, and retain the structured data that passes the consistency verification to determine the valid data.

[0051] The feature aggregation unit is used to calculate statistical features by grouping valid data according to the data source identifier, and generate an aggregation result containing the statistical features and the data source identifier;

[0052] The strategy generation unit is used to quantify the intensity of directional information transmission between data sources based on the aggregation results, construct an information flow network, identify the receiving nodes and sending nodes of information in the information flow network, generate a frequency reduction strategy for the receiving nodes and an enhancement strategy for the sending nodes, and distribute the frequency reduction strategy and enhancement strategy to the edge computing nodes.

[0053] The parameter adjustment unit is used to adjust the sampling parameters of the corresponding data source according to the frequency reduction strategy and the enhancement strategy.

[0054] A third aspect of the present invention provides an electronic device, comprising:

[0055] processor;

[0056] Memory used to store processor-executable instructions;

[0057] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0058] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0059] This method significantly improves data processing efficiency and resource utilization in edge computing environments. By performing frequency domain decomposition on the historical load sequences of edge nodes and extracting periodic features, it can accurately predict load fluctuation patterns. Then, during load increases, the original data stream is split into sub-streams for parallel processing, and during load decreases, backlogged data is merged and batch-processed. This dynamic scheduling mechanism effectively smooths out peak computing loads, avoiding processing delays or data loss caused by sudden high loads, while fully utilizing the idle computing power of nodes, achieving adaptive optimization of computing resources.

[0060] In this embodiment of the invention, by extracting the causal dependencies between structured data fields and verifying their global symbolic consistency, valid data can be accurately identified and retained, thereby significantly filtering out invalid or interfering information, reducing data redundancy in subsequent processing, improving data quality and reliability, and laying a solid foundation for generating accurate statistical features. Statistical features are calculated and aggregated results are generated by grouping data source identifiers, allowing for the quantitative characterization of the data source's output characteristics and facilitating systematic analysis. Based on the aggregated results, an information flow network is constructed and the intensity of directional information transmission between nodes is quantified, enabling clear identification of key information sending and receiving nodes in the network. Frequency reduction strategies are generated for receiving nodes, and enhancement strategies are generated for sending nodes. These strategies are then distributed to edge nodes to adjust the sampling parameters of the corresponding data sources, achieving closed-loop intelligent control of data acquisition frequency. This mechanism can dynamically optimize the data output of each data source according to actual information flow needs, effectively reducing unnecessary data acquisition and transmission while ensuring that key information is not lost, thus reducing the overall network bandwidth and storage resource consumption. Attached Figure Description

[0061] Figure 1 This is a flowchart illustrating intelligent data acquisition and edge computing methods.

[0062] Figure 2 This is a flowchart for time series causal analysis and consistency verification. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0065] Figure 1 This is a flowchart illustrating the intelligent data acquisition and edge computing method according to an embodiment of the present invention, as shown below. Figure 1 As shown, intelligent data acquisition and edge computing methods include:

[0066] Raw data streams are obtained from multiple data sources, and the raw data streams carry data source identifiers;

[0067] The historical load sequence of edge computing nodes is decomposed in the frequency domain to extract periodic features. The load fluctuation pattern is predicted based on the periodic features. During the load increase phase, the original data stream is split into sub-streams for parallel processing. During the load decrease phase, the backlog data stream is merged and batch processed. The original data stream is parsed to obtain structured data.

[0068] Extract the causal dependencies between fields in the structured data, identify the field combinations with causal dependencies, verify the global symbolic consistency of the field combinations through a multi-scale sliding window, and retain the structured data that passes the consistency verification to determine the valid data;

[0069] Calculate statistical features for valid data grouped by data source identifier, and generate aggregated results containing statistical features and data source identifier;

[0070] Based on the aggregation results, the intensity of directional information transmission between data sources is quantified, an information flow network is constructed, the receiving nodes and sending nodes of information in the information flow network are identified, a frequency reduction strategy is generated for the receiving nodes and an enhancement strategy is generated for the sending nodes, and the frequency reduction strategy and enhancement strategy are distributed to the edge computing nodes.

[0071] The sampling parameters of the corresponding data source are adjusted according to the frequency reduction strategy and the enhancement strategy.

[0072] In one optional embodiment, the historical load sequence of the edge computing node is decomposed in the frequency domain to extract periodic features. Based on these periodic features, the load fluctuation pattern is predicted. During the load increase phase, the original data stream is split into sub-streams for parallel processing. During the load decrease phase, the backlogged data stream is merged and batch-processed. The original data stream is then parsed to obtain structured data, including:

[0073] Perform frequency domain transformation on the historical load sequence, extract the frequency component with the largest amplitude as the load cycle feature, and record the cycle length and phase information corresponding to the frequency component.

[0074] Project the current load value onto the phase space of the load cycle characteristics, calculate the future load change slope corresponding to the current phase position, determine the load rising phase when the future load change slope is greater than zero, and determine the load falling phase when the future load change slope is less than zero and the backlog queue length exceeds the preset capacity.

[0075] During the load increase phase, the data dependencies between fields in the original data stream are analyzed. Fields that are directly dependent are grouped into the same field group, and field groups without dependencies are split into independent sub-streams. Each independent sub-stream is assigned a parsing thread for parallel processing, and each parsing thread outputs a structured fragment carrying the field group identifier.

[0076] During the load reduction phase, the data streams in the backlog queue are grouped according to the data source identifier, the data streams from the same data source are merged into batch processing units, and batch parsing is performed on the batch processing units to obtain batch structured data.

[0077] Based on the field group identifier, each structured fragment is concatenated in order according to the original field dependencies. The concatenated structured fragments are then merged with the batch structured data to generate complete structured data.

[0078] In one specific implementation, the historical load sequence of the edge computing node is subjected to frequency domain transformation, converting the time-domain load signal into a frequency-domain representation using a Fast Fourier Transform (FFT). The historical load sequence is typically collected from metrics such as CPU utilization, memory usage, or task queue length of the edge computing node over several past time windows, with sampling intervals set to seconds or minutes. The frequency domain transformation yields a spectrum, where the horizontal axis represents frequency and the vertical axis represents amplitude. All frequency components in the spectrum are traversed, and the frequency component with the largest amplitude is identified. This frequency component reflects the most significant periodic fluctuation pattern in the load sequence. The frequency value corresponding to this component is recorded, and the period length is calculated by the reciprocal of the frequency; for example, a frequency of 0.0167 Hz corresponds to a period length of 60 seconds. Simultaneously, the phase information of this frequency component is extracted, and this phase information is used to determine the starting position of the periodic fluctuation on the time axis.

[0079] The current load value is projected onto the phase space of the load cycle characteristics. Specifically, the time offset of the current moment relative to the start point of the cycle is calculated, and the remainder after dividing the time offset by the cycle length is taken to obtain the current phase position. Based on the extracted cycle length and phase information, a periodic load prediction model is constructed, which can be represented as a sine function superimposed with a DC component. The derivative of the prediction model at the current phase position is obtained to obtain the load change slope. When the derivative value is greater than zero, it indicates that the load is on an upward trend, and this is determined as the load rising phase. When the derivative value is less than zero, it indicates that the load is on a downward trend, but special processing for the load falling phase is only triggered when the backlog queue length exceeds a preset capacity threshold. The preset capacity threshold is usually set to 70% to 80% of the maximum queue capacity.

[0080] During the load increase phase, the data dependencies between fields in the raw data stream are analyzed. The raw data stream is typically in JSON or structured text format, containing multiple key-value pair fields. Data dependencies refer to the fact that parsing one field depends on the parsing result of another field; for example, the calculation of field B requires the value of field A as an input parameter. A field dependency graph is built through static analysis or runtime tracing, where nodes represent fields and directed edges represent dependencies. Fields with direct dependencies are grouped into the same field group; for example, if there is a dependency chain A→B→C between fields A, B, and C, they are grouped into field group G1. For fields or field groups without dependencies, they are split into independent sub-streams; for example, if there is no dependency between field group G1 and field group G2, G1 becomes sub-stream 1, and G2 becomes sub-stream 2. A dedicated parsing thread is allocated to each independent sub-stream, with the number of threads dynamically determined based on the number of available CPU cores on the current edge computing node. Each parsing thread independently executes the field parsing task, outputting structured fragments after parsing. Each structured fragment carries a field group identifier for subsequent fragment concatenation operations.

[0081] During the load reduction phase, data streams in the backlog queue are grouped according to their data source identifiers. The data source identifier is a tag carried by the original data stream, used to distinguish which sensor, device, or subsystem the data originates from. Multiple data streams from the same data source are merged to form batch processing units. The size of a batch processing unit can be dynamically adjusted based on the actual situation of the backlog queue, typically containing 10 to 100 data stream records. Batch parsing is performed on the batch processing units. Batch parsing reduces parser initialization overhead and context switching costs, improving processing efficiency. Specifically, batch parsing reads all data streams in the batch processing unit at once, uniformly performing lexical analysis, syntax parsing, and field extraction operations, outputting batch structured data. The batch structured data is a collection containing multiple structured records, each corresponding to one data stream from the original batch processing unit.

[0082] The structured fragments are concatenated sequentially according to their original field dependencies based on the field group identifiers. Specifically, this involves reading the structured fragments output by each parsing thread, extracting the field group identifiers from the fragments, and determining the concatenation order of each field group based on the field dependency graph. For example, if field group G1 needs to be concatenated before field group G2, the structured fragment corresponding to G1 is first written to the beginning of the final structured data, and then the structured fragment corresponding to G2 is appended to the subsequent field positions. During the concatenation process, boundary alignment between field groups needs to be handled to ensure that dependent fields can correctly reference the parsing results of preceding fields. The concatenated structured fragments are then merged with the batch structured data generated during the load reduction phase. The merging operation uses timestamp sorting or data stream sequence number sorting to ensure the temporal correctness of the final structured data. The merged data generates complete structured data containing all valid field information produced by both the parallel processing and batch processing phases.

[0083] In practical applications, edge computing nodes are deployed in industrial sites or IoT gateway devices. The historical load sequence collection period is typically the load records of the past 24 hours to one week. During frequency domain decomposition, window functions can be used to preprocess the historical load sequences, reducing the impact of spectral leakage on periodic feature extraction. Parallel processing during load increases fully utilizes the computing resources of multi-core CPUs, avoiding data processing delays during load peaks. Batch processing strategies during load decreases quickly clear the backlog queue during load troughs, preventing data loss due to queue overflow. The granularity of field group partitioning is dynamically adjusted according to the complexity of the data stream. For complex data streams containing a large number of fields, field groups are further subdivided into smaller subgroups to improve parallelism. The structured fragment splicing operation uses zero-copy technology to reduce memory copying overhead, and the merging of batch structured data and spliced ​​fragments uses a merge sort algorithm to ensure time complexity. Through the above adaptive load control mechanism and flexible data stream processing strategy, edge computing nodes can efficiently complete the parsing task of the original data stream under dynamic load conditions, outputting high-quality structured data for subsequent analysis.

[0084] In one alternative embodiment, splitting a group of fields without dependencies into independent substreams includes:

[0085] Generate a unique sub-stream identifier for each independent sub-stream;

[0086] Extract the global sequence number of each data record in the original data stream, combine the global sequence number with the substream identifier to generate a composite identifier, and append the composite identifier to the corresponding data record in the independent substream;

[0087] Record the field position index of the field group corresponding to each independent sub-stream in the original data stream, and store the field position index with the sub-stream identifier;

[0088] The independent sub-streams carrying composite identifiers are assigned to the parsing thread for parallel processing;

[0089] The structured fragments output by each parsing thread retain composite identifiers and field position indexes.

[0090] In one specific implementation, after the edge computing node receives the raw data stream, it analyzes the relationships between the fields in the data stream. By detecting the dependencies between fields, it identifies which groups of fields do not have causal dependencies. When multiple groups of fields are detected to be independent and undependent, these undependent groups of fields can be split into independent substreams, thereby enabling parallel processing to improve data parsing efficiency.

[0091] To achieve effective management and subsequent reorganization of independent sub-streams, each independent sub-stream needs to be assigned a unique identifier upon generation. This identifier can be generated using a combination of timestamp and node number. Specifically, when splitting field groups, the edge computing node obtains the current system time in milliseconds and its own unique number. The timestamp and node number are then combined using a specific encoding rule to form the sub-stream identifier. This sub-stream identifier remains unique within the current edge computing node's processing cycle, ensuring that there are no identifier conflicts between different independent sub-streams.

[0092] Each data record in the original data stream is assigned a global sequence number upon entering the edge computing node. This global sequence number is maintained by the data stream management module and is assigned to each newly arriving data record in an incremental manner. The purpose of the global sequence number is to record the arrival order of data records in the original data stream, ensuring that the original temporal sequence can be restored during subsequent data reassembly. After an independent sub-stream is generated, the global sequence number needs to be combined with the sub-stream identifier. The combination method can be string concatenation or encapsulation in a structured object. For example, the sub-stream identifier can be used as a prefix, the global sequence number as a suffix, and connected by a specific separator to form a composite identifier. The composite identifier contains both sub-stream attribution information and data record temporal information, providing the necessary indexing basis for merging data fragments after parallel processing.

[0093] After generating the composite identifier, it needs to be appended to the corresponding data record in the independent substream. Specifically, this involves adding a composite identifier field to the metadata area of ​​the data record. This field does not affect the actual content of the data record; it serves only as auxiliary information during processing. The appending operation can be achieved by inserting an additional identifier attribute into the data record structure, or by maintaining a separate mapping table to associate the data record with the composite identifier. The former directly embeds the composite identifier into the data record, resulting in higher access efficiency but increasing the storage overhead of the data record; the latter maintains the association through an external mapping table, keeping the data record structure unchanged, but requiring additional table lookup operations for access. In scenarios where edge computing resources are limited, the direct embedding method is usually chosen to reduce table lookup overhead.

[0094] During field group splitting, it's necessary to record the position information of the field groups corresponding to each independent sub-stream within the original data stream. Data records in the original data stream typically contain multiple fields, each with a fixed position index. For example, if a data record contains ten fields with position indices from 0 to 9, and a field group corresponding to an independent sub-stream contains the 2nd, 5th, and 8th fields, then the field position indices for that field group are the set {2, 5, 8}. Recording these field position indices allows the parsing thread to parse only the fields at the corresponding positions when processing independent sub-streams, skipping the parsing of other fields and reducing computational overhead. Simultaneously, the field position indices also provide a basis for subsequent data reassembly and field repositioning.

[0095] Field position indexes and sub-stream identifiers need to be associated and stored. A key-value pair approach can be used, with the sub-stream identifier as the key and the set of field position indexes as the value, stored in an in-memory hash table or a persistent configuration file. When the parsing thread receives an independent sub-stream, it can query the corresponding field position index using the sub-stream identifier to determine which fields need to be parsed. This associative storage mechanism decouples sub-streams from field positions, facilitating dynamic adjustments to the field group partitioning strategy.

[0096] Independent substreams carrying composite identifiers need to be allocated to parsing threads for parallel processing. Edge computing nodes typically maintain a parsing thread pool, the size of which is dynamically configured based on the number of CPU cores and memory capacity of the node. The allocation of independent substreams can employ round-robin, random, or load balancing strategies. The round-robin strategy allocates independent substreams sequentially to each parsing thread in a fixed order; it is simple to implement but may lead to uneven load distribution. The random strategy randomly selects idle threads for allocation; it offers better load balancing than round-robin but increases the overhead of random number generation. The load balancing strategy monitors the queue length or processing time of each parsing thread in real time, prioritizing the allocation of independent substreams to threads with lighter loads; it maximizes the utilization of computing resources but has higher implementation complexity. In actual deployment, the appropriate allocation strategy should be selected based on the characteristics of the data stream and the node's performance.

[0097] After receiving an independent sub-stream, the parsing thread parses the fields based on their position indices. The parsing process includes field type identification, data format conversion, and numerical range validation. For example, if a field contains temperature sensor data, the parsing thread first identifies it as a floating-point number, then converts the byte stream to a floating-point number, and finally checks if the value is within a reasonable range. After parsing, a structured fragment is generated, containing the parsed field data along with an additional composite identifier and field position index.

[0098] Each parsing thread retains a composite identifier and field position index in the structured fragments output to ensure that subsequent data merging operations can accurately restore the structure of the original data stream. The global sequence number in the composite identifier is used to sort the structured fragments, ensuring that the order of the merged data is consistent with the original data stream; the sub-stream identifier in the composite identifier is used to distinguish the parsing results of different field groups to avoid data confusion; and the field position index is used to place the parsed field data in the correct position to restore the complete data record structure.

[0099] After parallel processing is complete, the structured fragments output by each parsing thread need to be merged. The merging module first sorts all structured fragments according to their global sequence numbers, then matches structured fragments belonging to different field groups of the same original data record based on the sub-stream identifier, and finally fills the data of each field group into the corresponding positions according to the field position index to reconstruct the complete structured data record. Through this mechanism, the original data stream, after being split into independent sub-streams and processed in parallel, can be accurately restored to structured data, which improves processing efficiency while ensuring data integrity and temporal consistency.

[0100] This splitting and parallel processing mechanism is particularly suitable for high-throughput edge computing scenarios. In the Industrial Internet of Things (IIoT), sensor devices continuously generate large amounts of data streams with numerous fields and complex dependencies between some fields. By identifying fields without dependencies and splitting them into independent substreams, the multi-core processing capabilities of edge computing nodes are fully utilized, significantly reducing data parsing latency and meeting the needs of real-time data processing.

[0101] like Figure 2 The diagram shows the flowchart for time series causal analysis and consistency verification.

[0102] In one optional embodiment, causal dependencies between fields in the structured data are extracted, combinations of fields with causal dependencies are identified, and the global symbolic consistency of the field combinations is verified using a multi-scale sliding window. Structured data that passes the consistency verification is retained to determine valid data, including:

[0103] Extract time series data from each field of structured data;

[0104] A time offset scanning matrix is ​​constructed for the time series data of each field. The time position of the second field relative to the first field is gradually adjusted within a preset time offset range. The change in conditional entropy between the first field and the second field is calculated at each time offset position. The time offset corresponding to the minimum value of the change in conditional entropy is recorded to determine the optimal time delay. When the optimal time delay is positive, the first field is determined to be the cause field and the second field is determined to be the effect field. The cause field, the effect field and the optimal time delay are recorded to form a causal dependency triple.

[0105] Aggregate causal dependency triples that have a common cause field or a common effect field into a field combination;

[0106] Perform time alignment verification on each field combination, generate a delay compensation sequence based on the optimal delay in the causal dependency triple, calculate the global symbol consistency index through a multi-scale sliding window, and mark the field combination as consistent when the global symbol consistency index exceeds the preset consistency threshold;

[0107] Extract the structured data corresponding to the field combinations that pass consistency to determine valid data.

[0108] In one specific implementation, in edge computing scenarios, the structured data generated by various sensors and actuators often includes multiple measurement fields such as temperature, pressure, flow rate, and vibration amplitude. These fields may have physical causal relationships; for example, an increase in temperature may lead to an increase in pressure, or a change in valve opening may cause flow fluctuations. To accurately identify these causal dependencies and filter out invalid data, in-depth analysis of the structured data is required.

[0109] When extracting time-series data for each field from structured data, each field is arranged in timestamp order to form an independent numerical sequence. Assume the structured data contains fields... The time series corresponding to each field is denoted as Here, the subscript i represents the field number, and n represents the number of sampling points. During the extraction process, timestamp alignment is maintained to ensure that data from different fields at the same time are in the same index position.

[0110] A time-offset scan matrix is ​​constructed to detect temporal causal relationships between fields. For the first field... With the second field Within the preset time offset range Within, with a fixed step size Gradually adjust the time position of the second field relative to the first field. Time offset. Values For each time offset Shift the time series data of the second field so that its k-th data point is aligned with the k-th data point of the first field. Align the data points.

[0111] Calculate the change in conditional entropy at each time offset. Conditional entropy Measure in the known first field In the case of the second field The uncertainty is addressed by first discretizing the numerical ranges of the two fields into bins, mapping continuous values ​​to a finite number of intervals. The joint probability distribution is then calculated. With marginal probability distribution Conditional entropy is calculated using a probability distribution. The change in conditional entropy is defined as follows: ,in Indicates the elapsed time offset The first field after. When When the value is large, it indicates that the first field has a strong predictive ability for the second field, and there is a potential causal relationship between the two.

[0112] The time offset at which the change in conditional entropy reaches its minimum value is recorded as the optimal time delay. Iterate through all time offsets to find the one that makes... Reaching the maximum value The value is the optimal delay. The sign of the optimal delay has a clear causal indication. When When the first field changes, it indicates that the second field needs to be shifted in the future direction to achieve the strongest association with the first field, meaning that the change in the first field occurred before the change in the second field. In this case, the first field is determined to be the cause field and the second field to be the effect field. Conversely, when the first field changes, the second field is determined to be the effect field. In this case, the second field is the cause field, and the first field is the effect field. Record the cause field identifier. Fruit end field identifier and optimal latency Constructing a causal dependency triple .

[0113] Perform the aforementioned time offset scan and conditional entropy calculation on all pairs of fields to obtain multiple causal dependency triples. Triples with a common cause field or a common effect field are aggregated into field combinations. For example, triples... and Common factor field Aggregate into field combinations For example, triples and Having common fruit end fields Aggregate into field combinations Aggregation operations are used to integrate scattered pairwise causal relationships into multivariate causal network fragments.

[0114] Perform time-series alignment verification on each field combination to examine the stability of causal relationships over a global time scale. Based on the optimal latency in the causal dependency triplet... Generate a delay compensation sequence. For each causal edge in the field combination. , will the fruit end field Time series shifted in the past Each time step, aligning it with the factor field. The causal response time is aligned. The compensated result-end field sequence is denoted as... .

[0115] A multi-scale sliding window method is used to calculate the global sign consistency index. Sign consistency focuses on the directional matching of trends, rather than the exact correspondence of numerical values. Multiple window scales are set. These correspond to short-term, medium-term, and long-term time spans, respectively. (Regarding window scales...) Divide the time series into segments of length [length missing]. A continuously sliding window with a window step size set to To ensure complete coverage, within each window, the difference sequence of the due-end fields is calculated. Difference sequence with the compensation consequence field Extract the sign of the difference sequence, labeling positive values ​​as +1, negative values ​​as -1, and zero values ​​as 0. Count the number of time points with the same sign within the window. Total window length The ratio of the values ​​yields the sign consistency within the window. .

[0116] The signed consistency of all windows is weighted and averaged, with the weights inversely proportional to the window size. Shorter windows are given higher weights to capture local fluctuations. Global signed consistency index. The calculation is a weighted sum of the consistency across all scale windows. When Exceeding the preset consistency threshold When the threshold is set to 0.7 to 0.85, the combination of fields is marked as consistent. The threshold setting needs to balance the strictness of causality with the fault tolerance of the data. Too high a threshold may exclude true but noisy causal relationships, while too low a threshold may introduce spurious associations.

[0117] Structured data corresponding to field combinations that failed consistency verification were considered to contain measurement anomalies, sensor malfunctions, or external interference and were not included in subsequent analysis. Structured data corresponding to field combinations that passed consistency verification were extracted. These data maintained inherent causal logic consistency over time, reflecting the evolutionary patterns of real physical processes. The verified data were recorded with their timestamps, field values, and field combination identifiers, forming a valid dataset. This valid dataset eliminated outliers with broken causal relationships or logical conflicts, providing a high-quality data foundation for subsequent statistical analysis and decision optimization.

[0118] In practical deployments, causal dependency identification and consistency verification are performed in a streaming manner on edge computing nodes. A fixed-length sliding buffer is maintained to store the most recent time-series data, triggering incremental updates each time new data arrives. Conditional entropy calculation uses an online algorithm to progressively update the probability distribution estimate, avoiding repeated scanning of historical data. The consistency index of the multi-scale window is updated quickly using a recursive formula, keeping the computational complexity within an acceptable range. Through these optimization measures, causal dependency analysis can adapt to the real-time requirements and resource-constrained characteristics of edge computing.

[0119] In an optional embodiment, time-series alignment verification is performed on each field combination. A delay compensation sequence is generated based on the optimal delay in the causal dependency triples. A global symbolic consistency index is calculated using a multi-scale sliding window. When the global symbolic consistency index exceeds a preset consistency threshold, the field combination is marked as consistent, including:

[0120] Based on the optimal time delay in the causal dependency triplet, the time series of the effect-end field is forward-shifted to generate a time delay compensation sequence, and the time series of the cause-end field is used as the reference baseline sequence.

[0121] Multiple sliding windows of different lengths are set, and each sliding window is synchronously slid over the reference sequence and the time delay compensation sequence. The rate of change of the values ​​of the reference sequence and the time delay compensation sequence within the sliding window is calculated.

[0122] For each sliding window, count the number of times the sign of the rate of change of values ​​is consistent within the sliding window, calculate the proportion of the number of times the sign of the rate of change of values ​​is consistent to the length of the sliding window, and determine the local sign matching degree of the sliding window.

[0123] The local symbol matching degree of all sliding windows is weighted and summed according to the weight coefficient corresponding to the sliding window length to obtain the global symbol consistency index of the field combination.

[0124] When the global symbol consistency index exceeds the preset consistency threshold, the field combination is marked as consistent.

[0125] In one specific implementation, after confirming the existence of a causal dependency in the field combination, its symbolic consistency over a global time span is rigorously verified to ensure that the identified causal relationship remains stable and reliable over a long period. The core of this verification process lies in handling the inherent time lag effect in the causal relationship and capturing causal response patterns at different time granularities through multi-scale analysis methods.

[0126] For each combination of fields to be verified, the optimal delay parameter is extracted from the causal dependency triples obtained in the preceding steps. This optimal delay reflects the time interval required for a change in the cause field to propagate to the effect field. To eliminate the impact of this time misalignment on the symbol consistency analysis, a forward shift operation on the time axis is required for the time series of the effect field. Specifically, the optimal delay value is subtracted from the timestamp of each data point in the effect field's time series, causing the response time of the effect field to move forward on the time axis, thus aligning it with the action time of the cause field. The effect field's time series after this shift operation is the delay-compensated sequence. The data value corresponding to each time point in this sequence should logically respond to the change in the cause field at the same time. Meanwhile, the original time series of the cause field is kept unchanged and used as a reference baseline sequence for subsequent symbol comparison analysis.

[0127] To comprehensively evaluate the causal consistency performance of field combinations across different time scales, multiple sliding windows of varying lengths are established. These window lengths can be set as different proportions of the total time series length, such as 5%, 10%, 20%, and 40% of the total length, thus covering multiple time scales from short-term fluctuations to medium- to long-term trends. Shorter sliding windows capture immediate response relationships and frequently changing causal patterns between fields, while longer sliding windows focus on verifying the stability and persistence of causal relationships over longer periods. Each sliding window moves synchronously on the reference series and the time-delay-compensated series with the same starting position and sliding step size, ensuring that the time intervals truncated for both series are completely consistent at each window position.

[0128] At the current position of each sliding window, data segments of the reference sequence and the time-delay compensation sequence are extracted within that window range. For the window segment of the reference sequence, the numerical change between adjacent time steps is calculated, i.e., the value at the later time step minus the value at the previous time step, thus obtaining the rate of change of the value at each time step within the window. The same calculation process is applied to the corresponding window segment of the time-delay compensation sequence to generate a sequence of the rate of change of the value of that sequence within the window. The signs of these rates of change reflect the direction of change of the field values: positive values ​​indicate an increase in value, negative values ​​indicate a decrease in value, and zero values ​​indicate that the value remains unchanged.

[0129] For each time step within the window, the sign of the rate of change of the reference sequence at that time is extracted, and the sign of the rate of change of the time-delay compensation sequence at the same time is compared point by point. When the signs of the rate of change of the two sequences are the same at a certain time (i.e., both are positive, both are negative, or both are zero), they are considered to have consistent signs at that time; if the signs are opposite or one is zero while the other is non-zero, they are considered to have inconsistent signs. All time steps within the sliding window are traversed, and the total number of consistent signs is recorded. Since different sliding windows have different lengths, directly comparing the number of consistent signs lacks comparability, so this number needs to be normalized. The number of consistent signs within the sliding window is divided by the actual length of the sliding window, i.e., the total number of time steps contained in the window, to obtain the proportion of consistent signs. This proportion is between zero and one; the closer the value is to one, the stronger the consistency of the causal relationship at that window scale, i.e., the direction of change of the cause field and the direction of change of the effect field are highly synchronized after time alignment. This proportion is defined as the local sign matching degree of the sliding window.

[0130] After obtaining the local symbol matching degree of each sliding window, it is necessary to integrate the matching degree information at different scales into a single comprehensive evaluation index. To this end, a weight coefficient is assigned to each sliding window, which is related to the length of the sliding window. Longer sliding windows cover a longer time range, their statistical results are less affected by random noise, and they have a stronger explanatory power for the long-term stability of causal relationships; therefore, they can be assigned a higher weight. Conversely, shorter sliding windows, while reflecting short-term causal responses, are more susceptible to instantaneous fluctuations and can be assigned a relatively lower weight. The specific value of the weight coefficient can be determined based on the ratio of the sliding window length to the total length of the time series. For example, the weight coefficient can be set as the normalized value of the sliding window length, or other monotonically increasing function forms can be used. The local symbol matching degree of each sliding window is multiplied by its corresponding weight coefficient, and then the weighted local symbol matching degrees of all sliding windows are summed to obtain the global symbol consistency index of the field combination. This index comprehensively reflects the causal symbol consistency performance of the field combination across multiple time scales; a higher value indicates that the causal relationship exhibits stable directional correlation across the global time range and multiple time granularities.

[0131] A consistency threshold is set as the standard for determining whether a field combination passes consistency verification. The value of this threshold should be set based on the reliability requirements of causal relationships in the actual application scenario, typically between 0.6 and 0.9. A higher threshold implies a stricter verification standard, capable of filtering out field combinations with more significant and stable causal relationships, but leading to the filtering of some valid causal relationships; a lower threshold lowers the verification threshold, retaining more field combinations but increasing the risk of false positives. After calculating the global symbolic consistency index for a field combination, the index value is compared with the preset consistency threshold. If the global symbolic consistency index is significantly greater than the preset consistency threshold, the field combination is determined to have passed global symbolic consistency verification, marked as consistent, and included in the subsequent valid data range. Conversely, if the global symbolic consistency index does not meet the threshold requirement, the causal relationship of the field combination is considered to lack sufficient stability and reliability over the global time frame, marked as inconsistent, and removed from the valid data candidate set to avoid making subsequent data source control decisions based on unreliable causal relationships.

[0132] Through the aforementioned multi-scale temporal alignment verification and global symbol consistency evaluation mechanism, field combinations exhibiting stable causal response patterns across long time spans and multiple time granularities can be effectively identified and retained. This verification process not only eliminates the interference of causal delay on symbol comparison but also, through a multi-scale window weighted fusion strategy, balances the dual requirements of short-term response sensitivity and long-term trend stability. This ensures that the identified valid data possesses a high-quality causal logic foundation, providing reliable data input for subsequent information flow network construction and data source sampling strategy optimization. The entire verification process has clear computational steps and objective quantitative judgment criteria, enabling efficient execution on edge computing nodes and achieving dynamic quality control of large-scale real-time data streams.

[0133] In one optional embodiment, the intensity of directional information transmission between data sources is quantified based on the aggregation results, an information flow network is constructed, and the receiving and sending nodes of information in the information flow network are identified, including:

[0134] Time series alignment is performed on the statistical features of each data source in the aggregation results to extract the time evolution trajectory of the statistical features of each data source;

[0135] Cross-correlation analysis is performed on the time evolution trajectories of statistical characteristics of each data source. The maximum cross-correlation number of the time evolution trajectory of the first data source relative to the time evolution trajectory of the second data source and its corresponding time lag are calculated. When the maximum cross-correlation number exceeds the preset correlation threshold and the time lag is positive, the first data source is determined to be the information sending source and the second data source is determined to be the information receiving source. The maximum cross-correlation number is used as the directional information transmission strength of the data source pair.

[0136] An information flow network is constructed with data sources as nodes, and directed transmission edges are established in the information flow network for each information sending source and information receiving source pair, pointing from the information sending source to the information receiving source.

[0137] Assign edge weight attributes to each directed transmission edge in the information flow network, take the directional information transmission intensity of the data source pair corresponding to the directed transmission edge as the edge weight value of the directed transmission edge, and mark the time lag corresponding to the data source pair for each directed transmission edge in the information flow network as the transmission delay mark.

[0138] Calculate the in-degree and out-degree of each node in the information flow network, mark the nodes with an in-degree greater than their out-degree as information receiving nodes, and mark the nodes with an out-degree greater than their in-degree as information sending nodes.

[0139] In one specific implementation, after calculating statistical features by grouping data sources by identifier and generating aggregated results containing statistical features and data source identifiers, the information transmission relationship between the data sources is further analyzed. The statistical features of each data source in the aggregated results are then time-series aligned. Since the sampling frequencies and data reporting timestamps of different data sources differ, it is necessary to map the statistical features of each data source to a unified time reference. Specifically, the smallest sampling time interval among all data sources is selected as the reference time granularity. For data sources with larger sampling intervals, a linear interpolation method is used to generate aligned feature values ​​at the reference time point. After time alignment, the temporal evolution trajectory of the statistical features of each data source is extracted. This trajectory reflects the characteristic change trend of a specific data source within the observation time window.

[0140] After time alignment, cross-correlation analysis is performed on the time evolution trajectories of statistical features from each data source. Cross-correlation analysis measures the correlation between two time series at different time offsets and can reveal whether a lagged causal response relationship exists between the data sources. For any two data sources, their time evolution trajectories are denoted as follows: and Calculate the cross-correlation function ,in This represents the time lag. The cross-correlation function is defined as follows: Where N is the length of the time series, and These are the means of the two sequences, respectively. In actual calculations, the time lag is iterated over within a certain range, for example, from... arrive ,in Depending on the business scenario, the observation window length is typically set to 30%. For the first data source relative to the second data source, calculate the cross-correlation coefficients for all lags and identify the largest cross-correlation coefficient. and its corresponding time lag .

[0141] When the maximum cross-correlation coefficient exceeds a preset correlation threshold and the time lag is positive, it indicates that the change in the first data source precedes that of the second data source, suggesting a directed information transfer from the first to the second data source. In this case, the first data source is designated as the information sender, and the second data source as the information receiver. The preset correlation threshold is set based on data quality and noise levels, typically between 0.6 and 0.8. A positive time lag signifies that the feature changes in the first data source lead those in the second data source over time, implying a potential causal relationship. The maximum cross-correlation coefficient... This strength value serves as the directional information transmission strength for the data source pair, quantifying the closeness of the information transmission. For all possible data source pairings, the above cross-correlation analysis process is repeated to identify all information sender-receiver pairings that meet the conditions.

[0142] After obtaining the directional information transmission relationships of all data source pairs, an information flow network is constructed with the data sources as nodes. This network is a directed graph structure, where each node represents a data source, and the node attributes include the data source identifier and basic statistical information of that data source. For each identified information sending source and receiving source pair, a directed transmission edge is established in the information flow network from the information sending source to the information receiving source. The direction of the directed edge explicitly indicates the direction of information flow, i.e., information flows from the sending node to the receiving node. If data source A is the information sending source of data source B, then a directed edge is created in the network from node A to node B. This directed graph structure can intuitively show the propagation path and dependencies of information throughout the entire data acquisition system.

[0143] To more accurately characterize the strength of information transmission, each directed transmission edge in the information flow network is assigned an edge weight attribute. The strength of the directional information transmission between the data source pairs corresponding to each directed transmission edge is used as the edge weight value. The larger the edge weight value, the stronger the information correlation along the transmission path and the more significant the influence between the data sources. For example, if the maximum cross-correlation coefficient between data source A and data source B is 0.85, then the weight of the directed edge from A to B is 0.85. Furthermore, each directed transmission edge in the information flow network is labeled with the time lag corresponding to the data source pair as a transmission delay marker. The transmission delay marker records the time offset required for information to propagate from the sending source to the receiving source; this parameter is crucial for understanding the dynamic response characteristics of the system. Both the edge weight and transmission delay, two key parameters, are stored in the edge attributes simultaneously, facilitating subsequent analysis and strategy generation.

[0144] After constructing the complete information flow network, calculate the in-degree and out-degree of each node in the network. The in-degree of a node is defined as the number of directed edges pointing to that node, reflecting the strength of that data source's role as an information receiver. The out-degree of a node is defined as the number of directed edges emanating from that node, reflecting the strength of that data source's role as an information sender. Traverse all nodes in the information flow network and calculate the in-degree value of each node. and out-degree value Nodes with an in-degree greater than their out-degree are marked as information receiving nodes. These nodes primarily receive information from other data sources, and their data changes are more driven by external factors. Nodes with an out-degree greater than their in-degree are marked as information sending nodes. These nodes primarily output information to other data sources, and their data changes guide other parts of the system. Nodes with an in-degree equal to their out-degree can be marked as relay nodes or balance nodes, which both receive and send information.

[0145] In practical applications, such as multi-sensor collaborative acquisition systems in an Industrial Internet of Things (IIoT) environment, temperature data collected by temperature sensors may change before vibration data from vibration sensors. This is because increased equipment temperature causes mechanical components to expand, leading to changes in vibration characteristics. Cross-correlation analysis reveals that the temporal evolution trajectory of the temperature data source precedes that of the vibration data source, with a maximum cross-correlation coefficient of 0.78 and a time lag of 5 seconds. In this case, the node corresponding to the temperature sensor is marked as the information transmitting node, and the node corresponding to the vibration sensor is marked as the information receiving node. Based on this identification, sampling strategies can be adjusted accordingly. For information transmitting nodes, the sampling frequency can be appropriately increased to capture key changes, while for information receiving nodes, the sampling frequency can be decreased when there are no significant changes in the information transmitting nodes, thereby optimizing overall data acquisition efficiency.

[0146] The information flow network constructed using the above method not only reveals the static relationships between data sources, but more importantly, identifies dynamic causal links with temporal order. Edge weights quantify the strength of these relationships, transmission delay markers reveal response latency, and node in-degree and out-degree characteristics distinguish information roles. These structured network features provide a quantitative basis for subsequent sampling strategy generation, enabling edge computing nodes to dynamically adjust sampling parameters for different data sources based on the network's topology, achieving optimized resource allocation and intelligent management of data acquisition.

[0147] In one optional embodiment, generating a frequency reduction strategy for the receiving node and an enhancement strategy for the sending node, and then distributing the frequency reduction strategy and the enhancement strategy to the edge computing node includes:

[0148] Extract the incoming directed transmission edges of each receiving node, calculate the product of the edge weight value and the transmission delay mark to obtain the delay weighting strength, accumulate all delay weighting strengths to obtain the cumulative incoming information, map the cumulative incoming information to the frequency reduction amplitude coefficient, adjust the sampling period of the corresponding data source of the receiving node according to the frequency reduction amplitude coefficient, and generate the frequency reduction strategy.

[0149] Extract the outgoing directed transmission edges of each sending node, filter the edges whose weight values ​​exceed the preset strength threshold to form a key transmission edge set, count the number of edges in the key transmission edge set to obtain the impact coverage metric, extract the minimum value of the transmission delay marker in the key transmission edge set to obtain the primary response delay, multiply the impact coverage metric and the reciprocal of the primary response delay to obtain the enhancement score, map the enhancement score to the enhancement amplitude coefficient, and adjust the sampling frequency of the data source corresponding to the sending node according to the enhancement amplitude coefficient to generate the enhancement strategy;

[0150] The frequency reduction strategy and the enhancement strategy are packaged into a strategy instruction package according to the data source identifier and sent to the edge computing node.

[0151] In one specific implementation, after constructing the information flow network and identifying the receiving and sending nodes, a differentiated sampling adjustment strategy is generated based on the node characteristics. The receiving node, as the information aggregation point, often receives redundant information from multiple upstream data sources, and its sampling frequency can be appropriately reduced; the sending node, as the information output source, has a significant impact on downstream nodes, and its sampling quality needs to be maintained or improved.

[0152] The frequency reduction strategy generation process for receiving nodes is as follows: First, traverse all nodes marked as receiving nodes in the information flow network, and extract all incoming directed transmission edges for each receiving node. An incoming directed transmission edge is an edge in the information flow network that points to that node. Each edge carries two attributes: an edge weight value and a transmission delay flag. The edge weight value characterizes the intensity of information transmission along that edge; a larger value indicates more significant information transmission. The transmission delay flag represents the time interval required for information to be transmitted from the upstream node to the receiving node, measured in sampling periods or time units.

[0153] For each incoming directed transmission edge of the receiving node, its edge weight is multiplied by the transmission delay flag to obtain the delay-weighted strength. This calculation method can simultaneously reflect the strength and timeliness of information transmission; the shorter the delay and the greater the strength of the edge, the more significant its real-time impact on the receiving node. The delay-weighted strengths of all incoming directed transmission edges of the receiving node are accumulated to obtain the cumulative incoming information volume. This index comprehensively reflects the total information input load carried by the receiving node.

[0154] The frequency reduction amplitude coefficient is determined based on the accumulated incoming information. A mapping relationship is established to convert the accumulated incoming information into the frequency reduction amplitude coefficient. The mapping method can be a piecewise linear mapping or a nonlinear mapping function. For example, when the accumulated incoming information is below a first threshold, the frequency reduction amplitude coefficient is set to 1.0, indicating no frequency reduction; when the accumulated incoming information is between the first and second thresholds, the frequency reduction amplitude coefficient increases linearly between 1.0 and 1.5; when the accumulated incoming information exceeds the second threshold, the frequency reduction amplitude coefficient is set to 1.5, representing the maximum frequency reduction amplitude. The frequency reduction amplitude coefficient directly affects the sampling period of the data source corresponding to the receiving node. Multiplying the original sampling period by the frequency reduction amplitude coefficient yields the adjusted sampling period. For example, if the original sampling period of a receiving node is 10 seconds and the frequency reduction amplitude coefficient is 1.3, the adjusted sampling period is 13 seconds, achieving a reduction in sampling frequency. The adjusted sampling period, along with the data source identifier corresponding to the receiving node, is encapsulated to form the frequency reduction strategy.

[0155] The enhancement strategy generation process for sending nodes is symmetrical to the frequency reduction strategy. It iterates through all nodes marked as sending nodes in the information flow network, extracting all outgoing directed edges for each sending node. An outgoing directed edge is an edge pointing from that node in the information flow network, carrying both edge weight and transmission delay information.

[0156] From all outgoing directed edges of the sending node, edges with weight values ​​exceeding a preset strength threshold are selected to form a key transmission edge set. The preset strength threshold is set according to the actual application scenario, for example, 1.2 times the average or 1.5 times the median of all edge weight values. Retaining only edges exceeding this threshold allows focus on information transmission paths that significantly impact downstream nodes, avoiding weak information flow interference in strategy generation. The number of edges in the key transmission edge set is counted to obtain an impact coverage metric, which reflects how many downstream nodes the sending node significantly influences; a larger value indicates a wider information radiation range for the sending node.

[0157] Simultaneously, the minimum value of all transmission delay markers is extracted from the set of critical transmission edges to obtain the primary response delay. This metric characterizes the fastest response speed at which information from the sending node is transmitted to downstream nodes; a smaller delay indicates stronger timeliness of information transmission. The enhancement score is obtained by multiplying the impact coverage metric by the reciprocal of the primary response delay. The reciprocal operation results in a larger reciprocal value corresponding to a smaller primary response delay, thus assigning higher weight to timeliness in the product. The enhancement score comprehensively reflects the breadth and timeliness of the sending node's impact.

[0158] The enhancement score is hierarchically mapped and converted into an enhancement amplitude coefficient. The mapping rule can be set as follows: when the enhancement score is below the first score threshold, the enhancement amplitude coefficient is 1.0, maintaining the original sampling frequency; when the enhancement score is between the first and second score thresholds, the enhancement amplitude coefficient increases between 1.0 and 1.4; when the enhancement score exceeds the second score threshold, the enhancement amplitude coefficient is set to 1.4, representing the maximum enhancement amplitude. The enhancement amplitude coefficient directly affects the sampling frequency of the data source corresponding to the sending node. The original sampling frequency is multiplied by the enhancement amplitude coefficient to obtain the adjusted sampling frequency. For example, if the original sampling frequency of a sending node is 0.1 times per second and the enhancement amplitude coefficient is 1.3, the adjusted sampling frequency is 0.13 times per second, achieving an increase in sampling frequency. The adjusted sampling frequency, along with the data source identifier corresponding to the sending node, is encapsulated to form an enhancement strategy.

[0159] After generating the frequency reduction and enhancement strategies, these strategies need to be deployed to edge computing nodes for execution. The frequency reduction and enhancement strategies are grouped and packaged according to the data source identifier to form a strategy instruction package. The strategy instruction package adopts a structured format, including fields such as data source identifier, strategy type identifier (frequency reduction or enhancement), adjusted sampling parameters (sampling period or sampling frequency), and strategy effective timestamp. For cases where multiple strategies may exist simultaneously from the same data source, priority rules are used for merging; for example, enhancement strategies have higher priority than frequency reduction strategies, or strategies with larger adjustment magnitudes are selected.

[0160] The policy instruction packet is sent to the corresponding edge computing node via the communication network. After receiving the policy instruction packet, the edge computing node parses out the data source identifier and the adjusted sampling parameters, and updates its local sampling configuration table. The sampling configuration table records the current sampling period or sampling frequency of each data source, as well as the timestamp of the most recent sampling. Based on the updated sampling configuration, the edge computing node performs data collection according to the new sampling parameters when the next sampling period arrives, realizing dynamic adjustment of the data source sampling behavior.

[0161] To ensure the reliability of policy distribution, policy instruction packets can carry checksums or digital signatures. Edge computing nodes verify these upon receipt to prevent transmission errors or malicious tampering. Simultaneously, after applying the policy, edge computing nodes report the policy execution status back to the central management node, including policy reception time, application time, and execution result, forming a closed-loop management mechanism.

[0162] By differentiating and distributing the aforementioned frequency reduction and enhancement strategies, intelligent allocation of data acquisition resources is achieved. Receiving nodes reduce their sampling frequency due to receiving a large amount of redundant information, thus reducing the computational and storage overhead caused by repetitive data acquisition. Sending nodes, due to their significant impact on downstream nodes, improve sampling quality, ensuring the integrity and timeliness of data from key information sources. This mechanism enables the overall data acquisition system to optimize resource utilization efficiency and reduce the load pressure on edge computing nodes while ensuring information quality.

[0163] A second aspect of the present invention provides an intelligent data acquisition and edge computing system, comprising:

[0164] The data acquisition unit is used to acquire raw data streams from multiple data sources, wherein the raw data streams carry data source identifiers;

[0165] The load optimization unit is used to extract periodic features by frequency domain decomposition of the historical load sequence of edge computing nodes, predict the load fluctuation pattern based on the periodic features, split the original data stream into sub-streams for parallel processing during the load increase phase, merge the backlog data streams for batch processing during the load decrease phase, and parse the original data stream to obtain structured data.

[0166] The data validation unit is used to extract the causal dependencies between fields in structured data, identify field combinations with causal dependencies, verify the global symbolic consistency of field combinations through a multi-scale sliding window, and retain the structured data that passes the consistency verification to determine the valid data.

[0167] The feature aggregation unit is used to calculate statistical features by grouping valid data according to the data source identifier, and generate an aggregation result containing the statistical features and the data source identifier;

[0168] The strategy generation unit is used to quantify the intensity of directional information transmission between data sources based on the aggregation results, construct an information flow network, identify the receiving nodes and sending nodes of information in the information flow network, generate a frequency reduction strategy for the receiving nodes and an enhancement strategy for the sending nodes, and distribute the frequency reduction strategy and enhancement strategy to the edge computing nodes.

[0169] The parameter adjustment unit is used to adjust the sampling parameters of the corresponding data source according to the frequency reduction strategy and the enhancement strategy.

[0170] A third aspect of the present invention provides an electronic device, comprising:

[0171] processor;

[0172] Memory used to store processor-executable instructions;

[0173] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0174] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0175] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent data acquisition and edge computing method, characterized in that, include: Raw data streams are obtained from multiple data sources, and the raw data streams carry data source identifiers; The historical load sequence of edge computing nodes is decomposed in the frequency domain to extract periodic features. The load fluctuation pattern is predicted based on the periodic features. During the load increase phase, the original data stream is split into sub-streams for parallel processing. During the load decrease phase, the backlog data stream is merged and batch processed. The original data stream is parsed to obtain structured data. Extract the causal dependencies between fields in the structured data, identify the field combinations with causal dependencies, verify the global symbolic consistency of the field combinations through a multi-scale sliding window, and retain the structured data that passes the consistency verification to determine the valid data; Calculate statistical features for valid data grouped by data source identifier, and generate aggregated results containing statistical features and data source identifier; Based on the aggregation results, the intensity of directional information transmission between data sources is quantified, an information flow network is constructed, the receiving nodes and sending nodes of information in the information flow network are identified, a frequency reduction strategy is generated for the receiving nodes and an enhancement strategy is generated for the sending nodes, and the frequency reduction strategy and enhancement strategy are distributed to the edge computing nodes. The sampling parameters of the corresponding data source are adjusted according to the frequency reduction strategy and the enhancement strategy.

2. The method according to claim 1, characterized in that, The historical load sequences of edge computing nodes are decomposed in the frequency domain to extract periodic features. Load fluctuation patterns are predicted based on these periodic features. During load increases, the original data stream is split into sub-streams for parallel processing. During load decreases, the backlogged data streams are merged and batch-processed. The original data stream is then parsed to obtain structured data, including: Perform frequency domain transformation on the historical load sequence, extract the frequency component with the largest amplitude as the load cycle feature, and record the cycle length and phase information corresponding to the frequency component. Project the current load value onto the phase space of the load cycle characteristics, calculate the future load change slope corresponding to the current phase position, determine the load rising phase when the future load change slope is greater than zero, and determine the load falling phase when the future load change slope is less than zero and the backlog queue length exceeds the preset capacity. During the load increase phase, the data dependencies between fields in the original data stream are analyzed. Fields that are directly dependent are grouped into the same field group, and field groups without dependencies are split into independent sub-streams. Each independent sub-stream is assigned a parsing thread for parallel processing, and each parsing thread outputs a structured fragment carrying the field group identifier. During the load reduction phase, the data streams in the backlog queue are grouped according to the data source identifier, the data streams from the same data source are merged into batch processing units, and batch parsing is performed on the batch processing units to obtain batch structured data. Based on the field group identifier, each structured fragment is concatenated in order according to the original field dependencies. The concatenated structured fragments are then merged with the batch structured data to generate complete structured data.

3. The method according to claim 2, characterized in that, Splitting undependent field groups into independent sub-streams includes: Generate a unique sub-stream identifier for each independent sub-stream; Extract the global sequence number of each data record in the original data stream, combine the global sequence number with the substream identifier to generate a composite identifier, and append the composite identifier to the corresponding data record in the independent substream; Record the field position index of the field group corresponding to each independent sub-stream in the original data stream, and store the field position index with the sub-stream identifier; The independent sub-streams carrying composite identifiers are assigned to the parsing thread for parallel processing; The structured fragments output by each parsing thread retain composite identifiers and field position indexes.

4. The method according to claim 1, characterized in that, Extract causal dependencies between fields in structured data, identify field combinations with causal dependencies, verify the global sign consistency of field combinations using a multi-scale sliding window, and retain structured data that passes the consistency verification to determine valid data, including: Extract time series data from each field of structured data; A time offset scanning matrix is ​​constructed for the time series data of each field. The time position of the second field relative to the first field is gradually adjusted within a preset time offset range. The change in conditional entropy between the first field and the second field is calculated at each time offset position. The time offset corresponding to the minimum value of the change in conditional entropy is recorded to determine the optimal time delay. When the optimal time delay is positive, the first field is determined to be the cause field and the second field is determined to be the effect field. The cause field, the effect field and the optimal time delay are recorded to form a causal dependency triple. Aggregate causal dependency triples that have a common cause field or a common effect field into a field combination; Perform time alignment verification on each field combination, generate a delay compensation sequence based on the optimal delay in the causal dependency triple, calculate the global symbol consistency index through a multi-scale sliding window, and mark the field combination as consistent when the global symbol consistency index exceeds the preset consistency threshold; Extract the structured data corresponding to the field combinations that pass consistency to determine valid data.

5. The method according to claim 4, characterized in that, Perform time-series alignment verification on each field combination, generate a delay compensation sequence based on the optimal delay in the causal dependency triples, calculate the global symbolic consistency index using a multi-scale sliding window, and mark the field combination as consistent when the global symbolic consistency index exceeds a preset consistency threshold. Based on the optimal time delay in the causal dependency triplet, the time series of the effect-end field is forward-shifted to generate a time delay compensation sequence, and the time series of the cause-end field is used as the reference baseline sequence. Multiple sliding windows of different lengths are set, and each sliding window is synchronously slid over the reference sequence and the time delay compensation sequence. The rate of change of the values ​​of the reference sequence and the time delay compensation sequence within the sliding window is calculated. For each sliding window, count the number of times the sign of the rate of change of values ​​is consistent within the sliding window, calculate the proportion of the number of times the sign of the rate of change of values ​​is consistent to the length of the sliding window, and determine the local sign matching degree of the sliding window. The local symbol matching degree of all sliding windows is weighted and summed according to the weight coefficient corresponding to the sliding window length to obtain the global symbol consistency index of the field combination. When the global symbol consistency index exceeds the preset consistency threshold, the field combination is marked as consistent.

6. The method according to claim 1, characterized in that, Based on the aggregation results, the intensity of directional information transmission between data sources is quantified, an information flow network is constructed, and the receiving and sending nodes of information in the information flow network are identified, including: Time series alignment is performed on the statistical features of each data source in the aggregation results to extract the time evolution trajectory of the statistical features of each data source; Cross-correlation analysis is performed on the time evolution trajectories of statistical characteristics of each data source. The maximum cross-correlation number of the time evolution trajectory of the first data source relative to the time evolution trajectory of the second data source and its corresponding time lag are calculated. When the maximum cross-correlation number exceeds the preset correlation threshold and the time lag is positive, the first data source is determined to be the information sending source and the second data source is determined to be the information receiving source. The maximum cross-correlation number is used as the directional information transmission strength of the data source pair. An information flow network is constructed with data sources as nodes, and directed transmission edges are established in the information flow network for each information sending source and information receiving source pair, pointing from the information sending source to the information receiving source. Assign edge weight attributes to each directed transmission edge in the information flow network, take the directional information transmission intensity of the data source pair corresponding to the directed transmission edge as the edge weight value of the directed transmission edge, and mark the time lag corresponding to the data source pair for each directed transmission edge in the information flow network as the transmission delay mark. Calculate the in-degree and out-degree of each node in the information flow network, mark the nodes with an in-degree greater than their out-degree as information receiving nodes, and mark the nodes with an out-degree greater than their in-degree as information sending nodes.

7. The method according to claim 1, characterized in that, Generating a frequency reduction strategy for the receiving node and an enhancement strategy for the transmitting node, and then distributing the frequency reduction and enhancement strategies to the edge computing nodes includes: Extract the incoming directed transmission edges of each receiving node, calculate the product of the edge weight value and the transmission delay mark to obtain the delay weighting strength, accumulate all delay weighting strengths to obtain the cumulative incoming information, map the cumulative incoming information to the frequency reduction amplitude coefficient, adjust the sampling period of the corresponding data source of the receiving node according to the frequency reduction amplitude coefficient, and generate the frequency reduction strategy. Extract the outgoing directed transmission edges of each sending node, filter the edges whose weight values ​​exceed the preset strength threshold to form a key transmission edge set, count the number of edges in the key transmission edge set to obtain the impact coverage metric, extract the minimum value of the transmission delay marker in the key transmission edge set to obtain the primary response delay, multiply the impact coverage metric and the reciprocal of the primary response delay to obtain the enhancement score, map the enhancement score to the enhancement amplitude coefficient, and adjust the sampling frequency of the data source corresponding to the sending node according to the enhancement amplitude coefficient to generate the enhancement strategy; The frequency reduction strategy and the enhancement strategy are packaged into a strategy instruction package according to the data source identifier and sent to the edge computing node.

8. An intelligent data acquisition and edge computing system, used to implement the method as described in any one of claims 1-7, characterized in that, include: The data acquisition unit is used to acquire raw data streams from multiple data sources, wherein the raw data streams carry data source identifiers; The load optimization unit is used to extract periodic features by frequency domain decomposition of the historical load sequence of edge computing nodes, predict the load fluctuation pattern based on the periodic features, split the original data stream into sub-streams for parallel processing during the load increase phase, merge the backlog data streams for batch processing during the load decrease phase, and parse the original data stream to obtain structured data. The data validation unit is used to extract the causal dependencies between fields in structured data, identify field combinations with causal dependencies, verify the global symbolic consistency of field combinations through a multi-scale sliding window, and retain the structured data that passes the consistency verification to determine the valid data. The feature aggregation unit is used to calculate statistical features by grouping valid data according to the data source identifier, and generate an aggregation result containing the statistical features and the data source identifier; The strategy generation unit is used to quantify the intensity of directional information transmission between data sources based on the aggregation results, construct an information flow network, identify the receiving nodes and sending nodes of information in the information flow network, generate a frequency reduction strategy for the receiving nodes and an enhancement strategy for the sending nodes, and distribute the frequency reduction strategy and enhancement strategy to the edge computing nodes. The parameter adjustment unit is used to adjust the sampling parameters of the corresponding data source according to the frequency reduction strategy and the enhancement strategy.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images