A method for edge data fragmentation acquisition and processing

Abstract

This invention relates to the field of data governance technology, and in particular to a method for edge data fragmentation and processing, comprising the following steps: S1: acquiring edge data based on an edge server; S2: preprocessing the acquired edge data; S3: fragmenting the preprocessed edge data; S4: storing the fragmented edge data. This invention, based on a data fragmentation algorithm, can acquire the latest edge data from the edge server in the shortest possible time, completing the fragmentation and acquisition operation. It corrects the problem that traditional optimization algorithms cannot effectively and quickly transmit data within a given data latency limit, solving the latency problem in the data acquisition process and ensuring data acquisition efficiency. Furthermore, the fragmented edge data is encoded, then scrambled and encrypted, and then encoded and scrambled and encrypted again in fragment units, ensuring the security of the acquired edge data.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, and in particular to a method for edge data fragmentation and processing. Background Technology

[0002] In the era of big data, with the explosive growth in business volume and the continuous expansion of application scale, relational databases are inherently prone to becoming system performance bottlenecks. Single-machine storage capacity, connection count, and processing power are all limited, leading to various bottlenecks in server CPU, disk I / O, and memory. When the data volume of a single table reaches 10 million or 100GB, due to the large number of query dimensions, even with the addition of slave databases and index optimization, performance still degrades significantly during many operations. The inherent "statefulness" of databases makes them less scalable than web and application servers. Furthermore, as the volume of generated data continues to grow, data centers need faster and smarter ways to process all this information. Data centers are embracing the idea of ​​edge computing as a solution because processing data closer to the source is more efficient than sending and receiving data from the cloud. Edge computing works by placing network and computing resources closer to the area they serve to reduce latency. However, the increasing volume of data also leads to a continuous surge in edge data; inappropriate collection methods can also cause data collection delays and reduce efficiency. Summary of the Invention

[0003] The purpose of this invention is to address the deficiencies mentioned in the background art by proposing an edge data segmentation acquisition and processing method.

[0004] The technical solution adopted in this invention is as follows:

[0005] A method for edge data fragmentation acquisition and processing is provided, including the following steps:

[0006] S1: Acquire edge data based on edge server collection;

[0007] S2: Perform preprocessing operations on the collected edge data;

[0008] S3: Perform fragmentation processing on the preprocessed edge data;

[0009] S4: Store the edge data acquired by the segmented acquisition in segments.

[0010] As a preferred technical solution of the present invention: in step S2, an identification correspondence table for edge data is pre-established.

[0011] As a preferred technical solution of the present invention: the preprocessing in S2 includes data cleaning and feature extraction processing of the collected edge data.

[0012] As a preferred technical solution of the present invention: in step S3, edge data is segmented based on the identifier correspondence table in step S2 using a data segmentation algorithm.

[0013] As a preferred embodiment of the present invention, the data sharding algorithm is as follows:

[0014] Calculate the information gain value E of edge data based on decision tree:

[0015]

[0016] Where n is the number of decision tree nodes; edge data features corresponding to values ​​E>1 are selected, R is a discrete feature of edge information with a range, and the probability distribution P(R=r) of R is given. i )satisfy:

[0017] P(R=r i ) = p i (i = 1, ..., m)

[0018] Where, r i p is the marginal data distribution function. i The probability distribution of marginal data;

[0019] The entropy H(R) of the discrete feature R is:

[0020]

[0021] The uncertainty of determining the edge data fragmentation category S based on conditional entropy H(S|R) is as follows:

[0022]

[0023] Wherein, H(S|R=r i ) is characterized by r i Conditional entropy when edge data is sharded into category S;

[0024] Calculate the mutual information in different marginal data to determine the importance Q of features to the hierarchy. k :

[0025]

[0026] Where t(;) represents the mutual information of the data, B j For edge datasets, U j V j For the feature set, r k It is a discrete feature;

[0027] Based on Q k A discrete feature value is used as the initial segmentation threshold. The calculated discrete feature dataset is divided into two classes, R1 and R2. The gray-level mean values ​​r1 and r2 of classes R1 and R2 are obtained, and the updated segmentation threshold is calculated based on the gray-level mean values ​​r1 and r2.

[0028]

[0029] in, For updating the sharding threshold;

[0030] Continuously obtain different categories and their average grayscale values ​​based on the different categories, and calculate new segmentation thresholds until the following conditions are met:

[0031]

[0032] Where μ is a preset value.

[0033] As a preferred technical solution of the present invention: the data sharding algorithm collects the selected sharding thresholds, and completes the sharding collection of edge data based on the collected sharding thresholds.

[0034] As a preferred technical solution of the present invention: In step S3, edge data is split based on the data sharding algorithm to obtain multiple shard edge data, and the edge data of each shard is encoded and then scrambled, and the edge data within the scrambled shard is then encrypted.

[0035] As a preferred technical solution of the present invention: in step S4, the encoding is performed in units of segments, and after scrambling the encoding again, each segment is encrypted.

[0036] As a preferred technical solution of the present invention: in step S4, the encrypted fragment edge data is stored in multiple fragment tables respectively.

[0037] As a preferred technical solution of the present invention: in step S4, multiple shard tables are evenly distributed to different memories for storage.

[0038] The edge data segmentation acquisition and processing method provided by this invention has the following advantages compared with the prior art:

[0039] This invention uses a data fragmentation algorithm to collect edge data, enabling the rapid acquisition of the latest edge data from edge servers and completing the edge data collection operation. Fragmenting the edge data overcomes the limitations of traditional optimization algorithms in achieving efficient and high-speed data transmission within given latency constraints, thus resolving latency issues and ensuring data collection efficiency. Furthermore, the fragmented edge data is encoded, scrambled, and encrypted, then encoded and scrambled again in fragment units, ensuring the security of the collected edge data. Attached Figure Description

[0040] Figure 1 This is a flowchart of a preferred embodiment of the present invention. Detailed Implementation

[0041] It should be noted that, unless otherwise specified, the embodiments and features described in this embodiment can be combined with each other. The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0042] Reference Figure 1 A preferred embodiment of the present invention provides a method for edge data segmentation acquisition and processing, comprising the following steps:

[0043] S1: Acquire edge data based on edge server collection;

[0044] S2: Perform preprocessing operations on the collected edge data;

[0045] S3: Perform fragmentation processing on the preprocessed edge data;

[0046] S4: Store the edge data acquired by the segmented acquisition in segments.

[0047] In step S2, an identifier correspondence table for edge data is pre-established.

[0048] The preprocessing in S2 includes data cleaning and feature extraction of the collected edge data.

[0049] In step S3, edge data is segmented based on the identifier correspondence table in step S2 using a data segmentation algorithm.

[0050] The specific data sharding algorithm is as follows:

[0051] Calculate the information gain value E of edge data based on decision tree:

[0052]

[0053] Where n is the number of decision tree nodes; edge data features corresponding to values ​​E>1 are selected, R is a discrete feature of edge information with a range, and the probability distribution P(R=r) of R is given. i )satisfy:

[0054] P(R=r i ) = pi (i = 1, ..., m)

[0055] Where, r i p is the marginal data distribution function. i The probability distribution of marginal data;

[0056] The entropy H(R) of the discrete feature R is:

[0057]

[0058] The uncertainty of determining the edge data fragmentation category S based on conditional entropy H(S|R) is as follows:

[0059]

[0060] Wherein, H(S|R=r i ) is characterized by r i Conditional entropy when edge data is sharded into category S;

[0061] Calculate the mutual information in different marginal data to determine the importance Q of features to the hierarchy. k :

[0062]

[0063] Where t(;) represents the mutual information of the data, B j For edge datasets, U j V j For the feature set, r k It is a discrete feature;

[0064] Based on Q k A discrete feature value is used as the initial segmentation threshold. The calculated discrete feature dataset is divided into two classes, R1 and R2. The gray-level mean values ​​r1 and r2 of classes R1 and R2 are obtained, and the updated segmentation threshold is calculated based on the gray-level mean values ​​r1 and r2.

[0065]

[0066] in, For updating the sharding threshold;

[0067] Continuously obtain different categories and their average grayscale values ​​based on the different categories, and calculate new segmentation thresholds until the following conditions are met:

[0068]

[0069] Where μ is a preset value.

[0070] The data sharding algorithm collects the selected sharding thresholds and, based on the collected sharding thresholds, completes the sharding collection of edge data.

[0071] In step S3, edge data is split based on the data sharding algorithm to obtain multiple shard edge data. Each shard edge data is then encoded and scrambled. Finally, the scrambled edge data within the sharded shard is encrypted.

[0072] In step S4, the encoding is performed in units of segments, and after another scrambling encoding process, each segment is encrypted.

[0073] In step S4, the encrypted shard edge data is stored in multiple shard tables.

[0074] In step S4, multiple shard tables are evenly distributed across different memory locations for storage.

[0075] In this embodiment, a data acquisition request is sent to the edge server, including the acquisition time, acquisition location, functional attributes of the entity, and entity status data acquired by the sensing device. The edge server receives the request, and the system acquires edge data. First, an identifier mapping table is established for the edge data. Then, the acquired edge data undergoes data cleaning and feature extraction processing. Based on the established identifier mapping table, the edge data is then segmented using a data segmentation algorithm.

[0076] When sharding edge data, use a decision tree to calculate the information gain value E of the edge data:

[0077]

[0078] Where n is the number of decision tree nodes; calculate the final E value, select the edge data features corresponding to values ​​E>1, and let R be a discrete feature of edge information with a range, and let the probability distribution P(R=r) be the probability distribution of R. i )satisfy:

[0079] P(R=r i ) = p i (i = 1, ..., m)

[0080] Where, r i p is the marginal data distribution function. iGiven the probability distribution of marginal data, the entropy H(R) of the discrete feature R is defined as:

[0081]

[0082] The conditional entropy, under the condition of known discrete features R, determines the uncertainty of the edge data fragmentation category S:

[0083]

[0084] Wherein, H(S|R=r i ) is characterized by r i Conditional entropy when edge data is sharded into category S;

[0085] After determining the edge data sharding categories, the mutual information in different edge data is calculated to determine the importance Q of the features to the hierarchy. k :

[0086]

[0087] Where t(;) represents the mutual information of the data, B j For edge datasets, U j V j For the feature set, r k Discrete features; based on Q k A discrete feature value is used as the initial segmentation threshold. The calculated discrete feature dataset is divided into two classes, R1 and R2. The gray-level mean values ​​r1 and r2 of classes R1 and R2 are obtained, and the updated segmentation threshold is calculated based on the gray-level mean values ​​r1 and r2.

[0088]

[0089] in, For updating the sharding threshold;

[0090] Repeat the above steps continuously, that is, continuously obtain different categories and their gray-scale averages based on the different categories, and calculate new segmentation thresholds, until the following condition is met:

[0091]

[0092] Where μ is a preset value.

[0093] The selected segmentation threshold is collected, and the edge data segmentation is completed based on the collected segmentation threshold.

[0094] It can collect the latest edge data from the edge server in the shortest possible time, completing the edge data collection operation. Furthermore, it performs edge data fragmentation collection, which corrects the problem that traditional optimization algorithms cannot effectively and quickly transmit data within a given data latency limit, solving the latency problem in the data collection process and ensuring data collection efficiency.

[0095] Based on the aforementioned data sharding algorithm, edge data is split into multiple shards. Each shard's edge data is encoded, scrambled, and then encrypted. This process ensures both the integrity and security of the edge data within each shard, and facilitates subsequent edge data acquisition and recovery. The data is then encoded again in shard units, scrambled once more, and then encrypted. The encrypted sharded edge data is then stored in multiple sharding tables. These sharding tables are evenly distributed across different memory locations for storage.

[0096] When edge data needs to be stored in memory, the fragments are decrypted and replaced according to the encoded data to restore the correct fragment storage order. Then, the edge data stored in the fragments is decrypted and replaced according to the encoded data to restore the collected edge data order.

[0097] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0098] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An edge data slice acquisition processing method, characterized in that: The method comprises the following steps: S1: collecting edge data based on an edge server; S2: performing preprocessing operation on the collected edge data; S3: performing sharding processing on the preprocessed edge data; S4: performing sharding storage on the edge data collected in the sharding; In S3, the sharding processing of the edge data is performed based on the identification corresponding table in S2 through a data sharding algorithm; The data sharding algorithm is specifically as follows: Calculate the edge data information gain value E based on a decision tree: wherein n is the number of decision tree nodes; E is the edge data feature corresponding to the selected value of E>1, R is a discrete feature with a range of edge information, and the distribution probability P(R=r i ) of R satisfies: P(R=r i ) = p i (i = 1,..., m) where r i is the marginal data distribution function, p i is the probability of the marginal data. The entropy H(R) of a discrete feature R is: Determine the uncertainty of S of the edge data sharding category based on the conditional entropy H(S|R): where H(S|R=r i ) is the conditional entropy of the feature r i with the edge data slice class S; computing the mutual information amount in different edge data, determining the importance degree Q of the feature pair hierarchy k : where t(; ) denotes the mutual information of the data, B j is the edge dataset, U j , V j is the feature set, r k is the discrete feature; Based on Q k one discrete feature value as an initial slice threshold The calculated slice discrete feature data set is divided into two categories R1 and R2, the gray mean values r1 and r2 of the two categories R1 and R2 are obtained, and the updated slice threshold is calculated based on the gray mean values r1 and r2: wherein, is the updated shard threshold; Continuously acquire different categories and their gray mean values based on the divided different categories, and calculate new sharding threshold values until the following conditions are met: Wherein, μ is a preset given value.

2. The edge data slice collection processing method of claim 1, wherein: In S2, an identification corresponding table of the edge data is established in advance.

3. The edge data slice harvesting processing method of claim 1, wherein: The preprocessing in S2 includes data cleaning and feature extraction processing on the collected edge data.

4. The edge data slice harvesting processing method of claim 1, wherein: The data sharding algorithm collects selected sharding threshold values, and completes the sharding collection of the edge data based on the collected sharding threshold values.

5. The edge data slice harvesting processing method of claim 1, wherein: In S3, the sharding processing of the edge data is performed based on the data sharding algorithm, a plurality of sharding edge data are obtained, and each piece of sharding edge data is encoded, then is shuffled, and then the sharding edge data in the shuffled sharding is encrypted.

6. The edge data slice harvesting processing method of claim 1, wherein: In S4, the sharding is encoded again, and then is shuffled and encoded again, and then each sharding is encrypted.

7. The edge data slice harvesting processing method of claim 1, wherein: In S4, the encrypted sharding is stored in a plurality of sharding tables respectively.

8. The edge data slice harvesting processing method of claim 1, wherein: In S4, the plurality of sharding tables are evenly distributed to different memories for storage.

Images