A real-time data processing method and system based on edge computing

Abstract

The present invention provides a real-time data processing method and system based on edge computing, which relates to the field of data processing technology. The present invention calculates the data validity period through a composite decision model of the physical sampling frequency constraints of terminal equipment and preset business rules, effectively balancing the signal restoration accuracy requirements and the urgency of business scenarios, and avoiding the timeout of critical data or the crowding out of non-critical data resources; improves the data transmission integrity in a bandwidth fluctuation environment through a dynamic window cutting mechanism driven by real-time network status; optimizes the edge computing power load distribution by combining routing decision mapping strategy and processing priority resource allocation; improves the reliability of terminal instructions based on hierarchical control based on historical data outlier purification and geometric distance measurement; forms a complete technical chain from data acquisition, adaptive transmission cutting, time-sensitive and accurate arbitration, routing resource coordination to closed-loop control feedback, and enhances the transmission efficiency, resource utilization accuracy and control stability of high-concurrency real-time data processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a real-time data processing method and system based on edge computing. BACKGROUND

[0002] Under the background of the rapid development of the Internet of Things and edge computing, large-scale terminal devices continuously generate high-density real-time data streams, and the Internet of Things system faces three core bottlenecks. First, the traditional fixed window data cutting mechanism cannot dynamically adapt to network bandwidth fluctuations, resulting in serious damage to data packet integrity in high-frequency sampling scenarios. Second, data validity period calculation relies only on a single business rule or device sampling constraint, lacks a dual-arbitration model of physical layer time efficiency characteristics and business rules, and causes overloading of high-value data or discarding of critical data. Third, routing decisions and computing resource allocation are managed separately, neither a quantitative mapping mechanism from network state snapshots to routing health degrees nor dynamic allocation of computing resources combined with data time efficiency priority. This results in unbalanced load of edge nodes. Fourth, control instruction generation relies on historical benchmark data containing noise, and the difference evaluation between structured results and benchmarks lacks a quantitative grading mechanism for geometric distance measurement, frequently causing miscontrol accidents. Although existing solutions partially optimize data transmission efficiency or resource utilization, they fail to overcome the technical bottlenecks of network adaptive cutting, dual-factor validity period arbitration, routing resource collaborative scheduling, and benchmark purification closed-loop control, which seriously restrict the improvement of precision and robustness of real-time systems.

[0003] Therefore, it is necessary to provide a real-time data processing method and system based on edge computing to solve the above technical problems. SUMMARY

[0004] To solve the above technical problems, the present application provides a real-time data processing method and system based on edge computing, which achieves the beneficial effects of optimizing the efficiency of real-time data transmission processing and improving the utilization rate.

[0005] The present application provides a real-time data processing method based on edge computing, comprising:

[0006] S1: The terminal device continuously collects raw data streams based on a pre-set physical sampling period, and the edge gateway receives the raw data streams and marks the device fingerprint and timestamp for the raw data streams to obtain marked data packets;

[0007] S2: Based on real-time network state data, the marked data packets are subjected to variable time window cutting to generate segmented data blocks containing network state snapshots and timestamp sequences;

[0008] S3: Based on the physical sampling frequency of the terminal device and the pre-set business rules, the data validity period of the segmented data blocks is calculated, and the processing priority is generated in combination with the bandwidth utilization rate;

[0009] S4: determining a routing decision policy based on the network state snapshot, the real-time bandwidth in the real-time network state data, and a preset decision rule mapping table, allocating computing resources based on a processing priority, and generating a structured result set by a resource scheduler according to the routing decision policy and the computing resources processing the segmented data blocks;

[0010] S5: obtaining a historical benchmark data set based on the device fingerprint, calculating a geometric distance measure between the structured result set and the historical benchmark data set, generating a hierarchical control instruction based on the geometric distance measure, and pushing the encrypted hierarchical control instruction to the terminal device.

[0011] Preferably, in step S2, the window length of the variable time window cutting is determined by the following steps:

[0012] calculating the network bandwidth standard deviation in the last continuous multiple preset bandwidth monitoring periods;

[0013] dividing the real-time bandwidth in the real-time network state data by the preset maximum theoretical network bandwidth, and multiplying the preset bandwidth weight factor to obtain a bandwidth utilization term;

[0014] multiplying the network bandwidth standard deviation by a preset bandwidth fluctuation adjustment coefficient to obtain a bandwidth fluctuation term;

[0015] adding the bandwidth utilization term and the bandwidth fluctuation term to obtain a time window decision value;

[0016] determining the window length based on the time window decision value and a preset window length constraint rule.

[0017] Preferably, in step S3, the calculation step of the data validity period includes:

[0018] obtaining the physical sampling frequency of the terminal device, dividing the physical sampling frequency by 2 to obtain a physical validity period;

[0019] identifying the data type of the segmented data block, and obtaining the service validity period of the segmented data block in combination with a preset service rule;

[0020] comparing the physical validity period and the service validity period, and taking the smaller value as the data validity period.

[0021] Preferably, in step S3, the calculation step of the processing priority includes:

[0022] extracting the bandwidth utilization from the real-time network state data;

[0023] calculating the remaining effective time ratio of the segmented data block based on the survival time of the segmented data block and the data validity period;

[0024] generating the processing priority based on the bandwidth utilization, the remaining effective time ratio, and a preset data value coefficient.

[0025] Preferably, in step S4, the step of determining the routing decision strategy comprises:

[0026] quantifying the network state snapshot into a routing health score through a preset decision rule mapping table;

[0027] extracting a real-time bandwidth from real-time network state data, and calculating a bandwidth adaptability coefficient in combination with the routing health score and a preset routing weight factor;

[0028] selecting the routing decision strategy based on the bandwidth adaptability coefficient.

[0029] Preferably, in step S4, the routing decision strategy comprises a target processing node type, a transmission path selection strategy and a disaster recovery mechanism.

[0030] Preferably, in step S4, the step of generating the structured result set comprises:

[0031] outputting a time dimension feature vector after correcting data clock offset based on the timestamp in the segmented data block;

[0032] querying a preset device database to obtain geographical coordinates according to the device fingerprint in the segmented data block, and constructing a spatial dimension distribution matrix;

[0033] extracting multi-dimensional feature values from the data payload of the segmented data block;

[0034] fusing the time dimension feature vector, the spatial dimension distribution matrix and the multi-dimensional feature values into a structured feature tree, and encapsulating the structured feature tree into a standardized structured result set.

[0035] Preferably, in step S5, the geometric distance metric is a weighted Euclidean distance between the structured result set and the historical reference data set.

[0036] Preferably, in step S5, the step of obtaining the historical reference data set comprises:

[0037] querying a preset device database according to the device fingerprint in the segmented data block to extract a historical running data set of the same data type as the segmented data block;

[0038] calculating the average value and the standard deviation of all data points in the historical running data set, and filtering out data points in the historical running data set that deviate from the average value by plus or minus 3 times the standard deviation, taking the filtered historical running data set as the historical reference data set.

[0039] The application provides a real-time data processing system based on edge computing, which is applied to a real-time data processing method based on edge computing and comprises:

[0040] A data identification module is configured to continuously collect original data streams based on a preset physical sampling period, and an edge gateway is configured to mark device fingerprints and timestamps for the original data streams after receiving the original data streams, to obtain marked data packets;

[0041] A dynamic cutting module is configured to perform variable time window cutting on the marked data packets based on real-time network state data, to generate segmented data blocks containing network state snapshots and timestamp sequences;

[0042] A resource decision module is configured to calculate data validity periods of the segmented data blocks based on a physical sampling frequency of a terminal device and preset service rules, and to generate processing priorities in combination with bandwidth utilization;

[0043] A routing scheduling module is configured to determine a routing decision strategy based on network state snapshots, real-time bandwidth in real-time network state data and a preset decision rule mapping table, to allocate computing resources based on the processing priorities, and to generate a structured result set by a resource scheduler based on the routing decision strategy and the computing resources processing the segmented data blocks;

[0044] A closed-loop control module is configured to obtain a historical reference data set based on the device fingerprints, to calculate a geometric distance measure between the structured result set and the historical reference data set, to generate hierarchical control instructions based on the geometric distance measure, and to push the encrypted hierarchical control instructions to the terminal device.

[0045] Compared with related technologies, the real-time data processing method and system based on edge computing provided by the present application has the following beneficial effects:

[0046] The present application calculates data validity periods by a composite decision model of a terminal device physical sampling frequency constraint and preset service rules, effectively balances signal restoration accuracy requirements and business scenario urgency, avoids timeout invalidation of key data or resource squeezing of non-key data, improves data transmission integrity in a bandwidth fluctuation environment by a real-time network state driven dynamic window cutting mechanism, optimizes edge computing power load distribution in combination with routing decision mapping strategies and processing priority resource allocation, improves terminal instruction reliability based on historical data outlier purification and geometric distance measure hierarchical control methods, and forms a complete technical chain from data collection and marking, adaptive transmission cutting, time-sensitive accurate arbitration, routing resource coordination to closed-loop control feedback, thereby significantly enhancing transmission efficiency, resource utilization accuracy and control instruction stability of a high-concurrency real-time data processing system in an Internet of Things scenario. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 A flowchart of the real-time data processing method based on edge computing of the present application;

[0048] Figure 2 A module structure diagram of the real-time data processing system based on edge computing of the present application. DETAILED DESCRIPTION

[0049] The application will be further described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are intended to be illustrative only and are not intended to limit the scope of the application. In addition, it should be noted that, for the sake of brevity, the figures have not necessarily been drawn to scale and that the specific embodiments delineated herein are shown by way of example only and not limitation. Furthermore, the embodiments and features of the application can be combined with each other, if not incompatible, to provide further embodiments.

[0050] It should also be noted that, for the sake of brevity, the figures have not necessarily been drawn to scale and that the specific embodiments delineated herein are shown by way of example only and not limitation. Furthermore, the embodiments and features of the application can be combined with each other, if not incompatible, to provide further embodiments. Before discussing the example embodiments in more detail, it should be noted that some of the example embodiments are described as processes depicted as flow diagrams. Although the processes are described in a particular sequential order, many of the processes can be performed in parallel, concurrently or in any order. In addition, the order of the processes can be re-arranged. The processes can be terminated when their operations are completed, but the processes can also have additional steps not included in the figure, which can also be performed after the operations of the processes are completed. The processes can correspond in part to method steps.

[0051] Embodiment One

[0052] An edge computing-based real-time data processing method, in the specific implementation process, as shown in Figure 1 A flowchart of an edge computing-based real-time data processing method is shown, which includes:

[0053] Step S1: The terminal device continuously collects the original data stream based on the preset physical sampling period, and the edge gateway marks the device fingerprint and timestamp for the original data stream after receiving the original data stream, to obtain a marked data packet.

[0054] In the implementation process, the terminal device first continuously collects the original data stream according to the preset physical sampling period, which is set to millisecond level accuracy according to the sensor response time and industrial control requirements. For example, the temperature sensor collects environmental data every 100 milliseconds, and the vibration sensor collects device mechanical parameters every 50 milliseconds. After the edge gateway receives the original data stream through the industrial Ethernet or 5G industrial private network, it immediately performs the device fingerprint marking operation. The device fingerprint is composed of a device unique identifier, a hardware type code, and a security hash code to form a hexadecimal encoding string, ensuring that the identity of each device is tamper-proof and traceable. At the same time, a satellite time module is used to add a nanosecond-level precision timestamp to the data stream, accurately recording the time node when the data reaches the gateway. When the edge gateway detects multiple concurrent data streams, it will generate a timestamp sequence in the order of reception to prevent clock drift of different device data. Finally, the device fingerprint and timestamp double-bonded marking data packet is output. The marking data packet is packaged in binary format, with the header containing the fingerprint identification area and the timestamp area, and the payload area storing the original data stream, forming a unified standardized transmission data unit to provide input basis for subsequent variable time window cutting.

[0055] Step S2: performing variable time window cutting on the marking data packet based on real-time network state data to generate segmented data blocks containing network state snapshots and timestamp sequences.

[0056] Specifically, in step S2, the window length of the variable time window cutting is determined by the following steps:

[0057] Calculate the network bandwidth standard deviation in the last continuous multiple preset bandwidth monitoring periods;

[0058] Divide the real-time bandwidth in the real-time network state data by the preset maximum theoretical network bandwidth, and multiply it by the preset bandwidth weight factor to obtain the bandwidth utilization term;

[0059] Multiply the network bandwidth standard deviation by the preset bandwidth fluctuation adjustment coefficient to obtain the bandwidth fluctuation term;

[0060] Add the bandwidth utilization term and the bandwidth fluctuation term to obtain the time window decision value;

[0061] Determine the window length based on the time window decision value and the preset window length constraint rule.

[0062] In the implementation process, the network bandwidth value sequence in the latest three continuous preset bandwidth monitoring periods is continuously monitored, each bandwidth monitoring period is a preset 200 millisecond time unit, the bandwidth standard deviation of the network bandwidth value sequence is calculated to accurately quantify the network fluctuation intensity; at the same time, the current real-time bandwidth value is extracted from the real-time network state data, which is divided by the preset maximum theoretical network bandwidth to obtain the original utilization ratio value, and then the original utilization ratio value is multiplied by the preset bandwidth weight factor, for example, the preset bandwidth weight factor is 0.8, to generate the bandwidth utilization ratio item, and the preset bandwidth weight factor is used to strengthen the influence of the current load on the window decision; the calculated bandwidth standard deviation is multiplied by the preset bandwidth fluctuation adjustment coefficient to form the bandwidth fluctuation item, for example, the preset bandwidth fluctuation coefficient is 0.5, and the preset bandwidth fluctuation coefficient is used to convert the statistical fluctuation value into a time window adjustment amount; the bandwidth utilization ratio item and the bandwidth fluctuation item are added to obtain a time window decision value; according to the preset window length constraint rule, dynamic adjustment is performed, for example, the preset minimum window length threshold is 1 second and the maximum window length threshold is 5 seconds, if the time window decision value is less than 1 second, 1 second is used as the cutting window, if it is greater than 5 seconds, 5 seconds is used, and when it is in the interval of 1 to 5 seconds, the decision value is directly taken as the actual window length, based on the dynamically calculated window length, the cutting operation is performed on the marked data packet to generate a segmented data block, and the segmented data block contains a network state snapshot and a timestamp sequence, wherein the network state snapshot records the real-time bandwidth utilization value, the bandwidth fluctuation item value and the standard deviation parameter at the cutting time, and the timestamp sequence completely inherits the original data stream mark timestamp and maintains a strict increasing order, ensuring that each segmented data block carries accurate network instantaneous state characteristics and device-level time sequence continuity.

[0063] Step S3: Based on the physical sampling frequency of the terminal device and the preset service rule, the data validity period of the segmented data block is calculated, and the processing priority is generated based on the bandwidth utilization ratio.

[0064] Specifically, in step S3, the calculation steps of the data validity period include:

[0065] The physical sampling frequency of the terminal device is obtained, and the physical validity period is obtained by dividing the physical sampling frequency by 2;

[0066] The data type of the segmented data block is identified, and the service validity period of the segmented data block is obtained in combination with the preset service rule;

[0067] The size of the physical validity period and the service validity period is compared, and the smaller value is taken as the data validity period.

[0068] Specifically, in step S3, the calculation steps of the processing priority include:

[0069] The bandwidth utilization ratio is extracted from the real-time network state data;

[0070] a remaining valid time ratio of the segmented data block is calculated based on the survived time of the segmented data block and the data valid period;

[0071] a processing priority is generated based on the bandwidth utilization ratio, the remaining valid time ratio and a preset data value coefficient.

[0072] In the implementation process, the physical valid period is calculated according to the physical sampling frequency of the terminal device, the physical sampling frequency refers to the number of times of collecting data per second of the terminal device, and the physical valid period calculation formula is 2 divided by the sampling frequency value, wherein the value 2 is derived from the minimum double period coverage principle required by the Nyquist sampling theorem, which ensures that the time window contains at least two sampling points to ensure the integrity of signal restoration; at the same time, the data type of the segmented data block is identified, for example, the data type includes three types of device state data, environmental monitoring data and safety alarm data, and the business valid period is obtained by querying the preset business rule mapping table to obtain the preset time limit value corresponding to different data types; the smaller value of the physical valid period and the business valid period is taken as the final data valid period after comparison; then the current bandwidth utilization ratio parameter is extracted from the real-time network state data, which represents the proportion value of the current network traffic to the maximum theoretical bandwidth; the remaining valid time ratio is calculated by combining the survived time of the segmented data block and the data valid period, the survived time is equal to the current time minus the data generation time, and the remaining valid time ratio is equal to the survived time divided by the data valid period; the value weight value corresponding to the data type is queried from the preset data value coefficient mapping table according to the preset data value coefficient mapping table, and the weighted combination of the bandwidth utilization ratio, the remaining valid time ratio and the data value coefficient is obtained, the formula is , to obtain the processing priority, wherein is the processing priority, is the bandwidth utilization ratio, is the remaining valid time ratio, is the data value coefficient, is the network load weight factor fixedly set to 0.6, is the time limit urgency weight factor fixedly set to 0.3, is the business value weight factor fixedly set to 0.1, and the calculation result is normalized to the value range of zero to one hundred. The processing priority value will directly determine the allocation gradient of subsequent calculation resources and affect the processing order of the data block.

[0073] Step S4: determining a routing decision strategy based on the network state snapshot, the real-time bandwidth in the real-time network state data and the preset decision rule mapping table, allocating calculation resources based on the processing priority, and generating a structured result set by the resource scheduler based on the routing decision strategy and the calculation resources to process the segmented data block.

[0074] Specifically, in step S4, the step of determining the routing decision strategy comprises:

[0075] quantify the network state snapshot into a routing health score through a preset decision rule mapping table;

[0076] extract a real-time bandwidth from real-time network state data, and calculate a bandwidth adaptability coefficient in combination with the routing health score and a preset routing weight factor;

[0077] select a routing decision strategy based on the bandwidth adaptability coefficient.

[0078] Specifically, in step S4, the routing decision strategy includes a target processing node type, a transmission path selection strategy, and a disaster backup mechanism.

[0079] Specifically, in step S4, the generation step of the structured result set includes:

[0080] output a time dimension feature vector after correcting a data clock offset based on a timestamp in the segmented data block;

[0081] query a preset device database to obtain geographic coordinates based on a device fingerprint in the segmented data block, and construct a spatial dimension distribution matrix;

[0082] extract multi-dimensional feature values from the data payload of the segmented data block;

[0083] fuse the time dimension feature vector, the spatial dimension distribution matrix, and the multi-dimensional feature values into a structured feature tree, and encapsulate the structured feature tree into a standardized structured result set.

[0084] In the specific implementation process, first, the network state snapshot is quantized into a routing health score through a preset decision rule mapping table, the preset decision rule mapping table is a two-dimensional matrix with network state snapshot parameters as keys, the parameters include a real-time bandwidth utilization value recorded at the cutting time, a bandwidth fluctuation item value, and a standard deviation, the routing health score is mapped in the value interval of 0 to 100 through linear weighting, 0 represents that the network is completely unavailable, and 100 represents an ideal state; at the same time, a current real-time bandwidth value is extracted from real-time network state data, a bandwidth adaptability coefficient is calculated in combination with the routing health score and a preset routing weight factor, for example, the routing weight factor is preset according to the type of industrial network, for example, the Ethernet is set to 0.8 or the wireless network is set to 0.5, and the bandwidth adaptability coefficient calculation formula is: wherein, is the bandwidth adaptability coefficient, is the real-time bandwidth, is the routing health score, The routing weight factor is based on a numerical value of a broadband adaptability coefficient. For example, when the broadband adaptability coefficient is greater than 1.5, the target processing node type is an edge computing node; when the broadband adaptability coefficient is less than 1.0, the target processing node type is a cloud computing center; and when the broadband adaptability coefficient is in the interval of 1.0 to 1.5, the target processing node type is a hybrid processing mode. The transmission path selection strategy is implemented for different node types. For example, the edge computing node adopts a three-hop shortest path algorithm based on topology hashing, and the cloud computing center adopts a border gateway protocol optimal path. The resource scheduler processes the segmented data block based on the routing decision strategy. The specific execution structure of the generation of the structured result set is as follows: first, a time dimension feature vector is output by using a least square clock offset correction algorithm according to a time stamp sequence in the segmented data block. The time dimension feature vector is arranged in order according to the millisecond level time stamp. The geographic coordinate longitude and latitude values are obtained by querying a preset device database by using a device fingerprint in the segmented data block. The device fingerprint is a hash string for uniquely identifying a terminal device. The device database stores a mapping relationship table of the device fingerprint and the geographic coordinate. A spatial dimension distribution matrix is constructed, which is a two-dimensional Gaussian distribution weight matrix with the geographic coordinate as an index. Multidimensional feature values are extracted from the data load of the segmented data block. The data load is a set of original collected sensor physical quantity values. The dimensional feature values include maximum value, minimum value, average value and standard deviation statistics. The time dimension feature vector, the spatial dimension distribution matrix and the multidimensional feature values are input into a feature fusion engine to generate a structured feature tree. The structured feature tree adopts a binary prefix tree index structure to store the space-time data relationship. Finally, the structured feature tree is encapsulated into a standardized structured result set encapsulation body. The encapsulation body format is a self-describing data object including a time sequence vector field, a spatial distribution matrix field and a feature value array field. The final output of the edge computing data processing is completed.

[0085] Step S5: Based on the device fingerprint, a historical reference data set is obtained, and a geometric distance measure between the structured result set and the historical reference data set is calculated. Based on the geometric distance measure, a hierarchical control instruction is generated and pushed to the terminal device after being encrypted.

[0086] Specifically, in step S5, the geometric distance measure is a weighted Euclidean distance between the structured result set and the historical reference data set.

[0087] Specifically, in step S5, the historical reference data set is obtained by:

[0088] According to the device fingerprint in the segmented data block, a preset device database is queried to extract a historical running data set of the same data type as the segmented data block;

[0089] The average value and the standard deviation of all data points in the historical running data set are calculated, and data points deviating from the average value by plus or minus 3 times the standard deviation are screened out, and the screened historical running data set is taken as a historical reference data set.

[0090] In the implementation process, in the closed-loop control stage of the edge computing data processing flow, historical running data set is retrieved from the preset device database based on the device fingerprint as a reference, the device fingerprint is a hexadecimal hash string used to uniquely identify the terminal device, and the device database associated with the device fingerprint stores the historical running data set of the same type of device; the specific retrieval process is to filter the same type of historical running data set in the device database according to the data type of the segmented data block, the data type is matched with the business rule mapping classification, which includes three standard categories of device state data, environment monitoring data and security alarm data; statistical purification operation is performed on the obtained historical running data set, first, the arithmetic mean of all data points in the historical running data set is calculated as the reference center value, and the standard deviation value of the entire data set is calculated as the dispersion measure index, the arithmetic mean is calculated by adding all data point values and dividing by the total number, and the standard deviation is calculated by using the same root mean square algorithm as the network bandwidth monitoring statistics; the average value and the standard deviation are used to construct a data point filtering rule, and the data points deviating from the average value by more than three standard deviations in the positive direction or more than three standard deviations in the negative direction are accurately screened out, and the three standard deviation boundaries constitute the statistical significant abnormality discrimination threshold; the data point set after purification is used as the final historical reference data set to participate in subsequent geometric distance measurement calculation; then the structured result set and the historical reference data set are weighted to obtain the geometric distance measurement, the geometric distance measurement formula is the weighted sum of the square of each feature dimension difference and then taking the square root, the feature dimensions include time dimension sequence vector, spatial dimension matrix coordinate value and feature dimension statistics three types of core dimensions; the weight coefficient of each dimension is preset according to the importance of the feature, for example, the weight of the time dimension of the security alarm type is set to 0.7, the weight of the space dimension is set to 0.2, and the weight of the feature dimension is set to 0.1, and the weight of the time dimension of the environment monitoring type is set to 0.3, the weight of the space dimension is set to 0.5, and the weight of the feature dimension is set to 0.2; the distance calculation process is to subtract the feature value of the corresponding dimension of the historical reference data set from each dimension feature value of the structured result set to obtain the difference, multiply the square of the difference by the preset dimension weight coefficient, and take the square root of the sum of all weighted square values to obtain the final distance scalar value, that is, the geometric distance measurement; a hierarchical control instruction is generated based on the geometric distance measurement, the hierarchical rule is set according to the preset distance threshold interval, for example, when the geometric distance measurement is less than one standard deviation, a maintenance operation instruction is generated without processing, an optimization adjustment instruction is generated in the one to three standard deviation interval to trigger device parameter fine tuning, and an emergency shutdown instruction is generated when the three standard deviations are exceeded to immediately interrupt the device operation; the control instruction is encrypted using the asymmetric key derived from the device fingerprint, for example, the encryption algorithm uses the block cipher mode of the national standard SM4, and the encryption process adds quantum random number enhanced anti-replay signature; finally, the encrypted instruction is pushed to the terminal device through the secure transmission channel of the edge gateway to execute the closed-loop control, and the operation state of the terminal device after executing the instruction is filled back to the device database as a new data point to complete the self-learning update cycle.

[0091] The working principle of the real-time data processing method based on edge computing provided by the application is as follows:

[0092] Firstly, in the aspect of data acquisition, the terminal device generates an original data stream based on a preset physical sampling period, and the edge gateway realizes the binding of the unique identification of the data source and the space-time reference by embedding the device fingerprint and the timestamp; then, in the aspect of data transmission, the size of the cutting window is dynamically adjusted based on the real-time network state, the network fluctuation intensity is quantified by calculating the bandwidth standard deviation, the window decision value is generated by combining the real-time bandwidth utilization, the integrity and the time sequence continuity of the high-frequency data stream in the complex network environment are ensured, a double-arbitration mechanism is introduced, on the one hand, the minimum effective period under the Nyquist constraint is calculated according to the physical sampling frequency, and on the other hand, the scenario-based time requirement is obtained by mapping the business rules, the minimum value of the two is taken as the upper limit of the data effective period, and the multi-dimensional processing priority is generated by combining the bandwidth load, the proportion of the remaining effective period and the data value coefficient, so that the dynamic matching of the resource demand and supply is realized; further, the network state snapshot is quantified as the routing health score, the broadband adaptability coefficient is calculated in cooperation with the real-time bandwidth, the target node type and the transmission path strategy are decided according to the broadband adaptability coefficient, and the core resources are allocated based on the priority value gradient; the clock offset correction, the geographic coordinate mapping and the multi-dimensional feature extraction technology are used to generate the structured result set of the space-time feature fusion; finally, the historical data set after the purification of the device fingerprint index is used to calculate the weighted Euclidean geometric distance to measure the deviation between the real-time state and the reference, the hierarchical encryption instruction is generated according to the distance threshold and is pushed to the terminal device, and the whole-process closed-loop system from the data acquisition, the dynamic transmission, the intelligent arbitration, the resource optimization to the accurate control is formed, so that the efficient transmission of the high-concurrency real-time data processing and the improvement of the resource utilization are realized.

[0093] Embodiment two

[0094] A real-time data processing system based on edge computing is applied to a real-time data processing method based on edge computing, and in the specific implementation process, as shown in Figure 2 Fig. 1 shows a module structure diagram of a real-time data processing system based on edge computing, which comprises:

[0095] The data identification module 100 is used for the terminal device to continuously acquire the original data stream based on the preset physical sampling period, and the edge gateway marks the device fingerprint and the timestamp for the original data stream after receiving the original data stream, and obtains the marked data packet;

[0096] The dynamic cutting module 200 is used for performing variable time window cutting on the marked data packet based on the real-time network state data, and generating the segmented data block containing the network state snapshot and the timestamp sequence;

[0097] The resource decision module 300 is used for calculating a data validity period of the segmented data block based on a physical sampling frequency of the terminal device and a preset service rule, and generating a processing priority in combination with a bandwidth utilization rate;

[0098] The routing scheduling module 400 is used for determining a routing decision strategy based on the network state snapshot, the real-time bandwidth in the real-time network state data and a preset decision rule mapping table, allocating a computing resource based on the processing priority, and generating a structured result set by the resource scheduler according to the routing decision strategy and the computing resource processing the segmented data block;

[0099] The closed-loop control module 500 is used for acquiring a historical benchmark data set based on the device fingerprint, calculating a geometric distance measure between the structured result set and the historical benchmark data set, generating a hierarchical control instruction based on the geometric distance measure, and pushing the encrypted hierarchical control instruction to the terminal device.

[0100] The working principle of the real-time data processing system based on edge computing provided by the application is as follows:

[0101] The data identification module 100 injects a device fingerprint and a precise time stamp into the original data stream collected by the terminal device at the edge gateway layer, establishes a traceable unique identity and a time benchmark; the dynamic cutting module 200 calculates a dynamic window length by real-time monitoring of a network bandwidth standard deviation and a current utilization rate, executes variable time window cutting according to the dynamic window length, and generates segmented data blocks carrying a network state snapshot and timing characteristics; the resource decision module 300 generates a data validity period based on the double constraints of the physical sampling frequency of the terminal device and the preset service rule, and calculates a gradient processing priority by fusing a bandwidth utilization rate, a data survival time-to-live ratio and a value weight coefficient; the routing scheduling module 400 quantizes the network state snapshot into a routing health degree index, calculates a broadband adaptability coefficient in cooperation with a real-time bandwidth, determines a target processing type, a transmission path strategy and a disaster recovery backup mechanism of an edge node or a cloud computing center according to the broadband adaptability coefficient, and allocates accurate computing core resources according to the priority value; the resource scheduler calls the allocated computing resources to execute a time-space data processing pipeline: first, correct the time stamp sequence to generate a time vector, then construct a spatial distribution matrix according to the device fingerprint searching geographical coordinates, and finally fuse into a structured feature tree and encapsulate and output; the closed-loop control module 500 indexes the device database based on the device fingerprint, retrieves a historical data set of the same type, purifies the benchmark data by using a three-standard-deviation rule, calculates a weighted Euclidean geometric distance between the current structured result set and the purified benchmark, generates a hierarchical control instruction according to a distance threshold, and pushes the encrypted hierarchical control instruction to the terminal device, thereby forming a full-process self-optimization system from data identification, network dynamic cutting, resource intelligent decision, routing optimization scheduling to closed-loop feedback control, and realizing high timeliness, high integrity and high reliability of real-time data processing in the whole chain in the edge computing scenario.

[0102] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the working examples illustrate the architecture, functionality, and operation of possible implementations of apparatuses (systems), methods and computer program products according to the present disclosure. In this regard, each flowchart and / or block diagram can represent a method, apparatus and / or computer program products according to the present disclosure. It will also be understood that each block of the flowchart and / or block diagrams and combinations of blocks in the flowchart and / or block diagrams can be implemented by computer program instructions such as program code. Figure 1 These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart and / or block diagram block(s).

[0103] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk memories, magnetic disk memories, magnetic tape memories, or any other medium readable by a computer and capable of carrying or storing data.

[0104] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusions, such that processes, methods, articles or devices that include a series of elements not only include those elements, but also include other elements not explicitly listed, or other elements inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device that includes the element.

Claims

1. A real-time data processing method based on edge computing, characterized in that: The data processing method comprises the following steps: S1: The terminal device continuously collects raw data streams based on the preset physical sampling period. After receiving the raw data streams, the edge gateway marks the raw data streams with device fingerprints and timestamps to obtain marked data packets. S2: Perform variable time window segmentation on the marked data packets based on real-time network status data to generate segmented data blocks containing network status snapshots and timestamp sequences; S3: Calculates the data validity period of the segmented data blocks based on the physical sampling frequency of the terminal device and preset business rules, and generates processing priorities based on bandwidth utilization; S4: Determine the routing decision strategy based on the network status snapshot, the real-time bandwidth in the real-time network status data, and the preset decision rule mapping table, allocate computing resources based on processing priority, and the resource scheduler generates a structured result set based on the routing decision strategy and computing resource processing segmented data blocks; S5: Obtain a historical benchmark dataset based on the device fingerprint, calculate the geometric distance measurement between the structured result set and the historical benchmark dataset, generate hierarchical control instructions based on the geometric distance measurement, and push them to the terminal device after encryption; In step S3, the step of calculating the processing priority includes: Extract bandwidth utilization from real-time network status data; Calculate the remaining valid time ratio of the segmented data block based on the survival time and data validity period of the segmented data block; Generate processing priority based on bandwidth utilization, remaining effective time ratio and preset data value coefficient.

2. A real-time data processing method based on edge computing according to claim 1, characterized in that: In step S2, the window length of the variable time window cutting is determined by the following steps: Calculate the network bandwidth standard deviation within the latest multiple consecutive preset bandwidth monitoring periods; The bandwidth utilization term is obtained by dividing the real-time bandwidth in the real-time network status data by the preset maximum theoretical bandwidth of the network and then multiplying the result by the preset bandwidth weight factor; Multiply the network bandwidth standard deviation by the preset bandwidth fluctuation adjustment coefficient to obtain the bandwidth fluctuation term; The bandwidth utilization term and the bandwidth fluctuation term are added together to obtain the time window determination value; The window length is determined based on the time window decision value and the preset window length constraint rules.

3. A real-time data processing method based on edge computing according to claim 2, characterized in that: In step S3, the step of calculating the data validity period includes: Get the physical sampling frequency of the terminal device and divide it by 2 to get the physical validity period. Identify the data type of the segmented data block and obtain the business validity period of the segmented data block based on the preset business rules; Compare the physical validity period and the business validity period, and take the smaller value as the data validity period.

4. A real-time data processing method based on edge computing according to claim 3, characterized in that: In step S4, the step of determining the routing decision strategy includes: Quantify the network status snapshot into a routing health score through a preset decision rule mapping table; Extract real-time bandwidth from real-time network status data and calculate the bandwidth adaptability coefficient by combining the route health score and the preset route weight factor; Select routing decision strategy based on broadband adaptability coefficient.

5. A real-time data processing method based on edge computing according to claim 4, characterized in that: In step S4, the routing decision strategy includes the target processing node type, the transmission path selection strategy and the disaster recovery backup mechanism.

6. A real-time data processing method based on edge computing according to claim 5, characterized in that: In step S4, the steps of generating a structured result set include: Based on the timestamps in the segmented data blocks, the time dimension feature vector is output after correcting the data clock offset; According to the device fingerprint in the segmented data block, the preset device database is queried to obtain the geographic coordinates and a spatial dimension distribution matrix is ​​constructed; Extracting multi-dimensional feature values ​​from the data payload of the segmented data blocks; The time dimension feature vector, space dimension distribution matrix and multi-dimensional feature value are fused into a structured feature tree, and the structured feature tree is encapsulated into a standardized structured result set.

7. A real-time data processing method based on edge computing according to claim 6, characterized in that: In step S5 , the geometric distance metric is the weighted Euclidean distance between the structured result set and the historical benchmark dataset.

8. A real-time data processing method based on edge computing according to claim 7, characterized in that: In step S5, the steps for obtaining the historical benchmark dataset are: According to the device fingerprint in the segmented data block, a preset device database is queried to extract a historical operation data set with the same data type as the segmented data block; Calculate the mean and standard deviation of all data points in the historical operation data set, and filter out data points in the historical operation data set that deviate from the mean by plus or minus 3 times the standard deviation. Use the filtered historical operation data set as the historical benchmark data set.

9. A real-time data processing system based on edge computing, characterized in that: The method for real-time data processing based on edge computing according to any one of claims 1 to 8 is applied, wherein the data processing system comprises: The data identification module is used for the terminal device to continuously collect the original data stream based on the preset physical sampling period. After the edge gateway receives the original data stream, it marks the device fingerprint and timestamp for the original data stream to obtain a marked data packet; A dynamic segmentation module is used to perform variable time window segmentation on the marked data packets based on real-time network status data to generate segmented data blocks containing network status snapshots and timestamp sequences; The resource decision module is used to calculate the data validity period of the segmented data blocks based on the physical sampling frequency of the terminal device and the preset business rules, and generate the processing priority based on the bandwidth utilization; The routing scheduling module is used to determine the routing decision strategy based on the network status snapshot, the real-time bandwidth in the real-time network status data and the preset decision rule mapping table, allocate computing resources based on processing priority, and generate a structured result set based on the routing decision strategy and computing resource processing segmented data blocks; The closed-loop control module is used to obtain a historical benchmark data set based on the device fingerprint, calculate the geometric distance measurement between the structured result set and the historical benchmark data set, generate hierarchical control instructions based on the geometric distance measurement, and push them to the terminal device after encryption.

Images