Real-time data distribution and intelligent caching method and system based on edge computing

Abstract

The invention provides a real-time data distribution and intelligent caching method and system based on edge computing, and belongs to the technical field of computers, and the method comprises the steps: 1, collecting an original data stream of terminal equipment, converting the original data stream into a structured data format, and generating a feature vector based on the structured data; 2, outputting a first shunting signal and a second shunting signal through a pre-trained shunting decision model according to the feature vector; 3, extracting a feature identifier based on the first shunt signal, obtaining the feature identifier and a corresponding historical access record, and determining a final priority score of the corresponding data in the edge cache; 4, managing an edge cache space according to the priority score, and generating a cache update record; and 5, performing security processing on the structured data, generating a global cache distribution topology, and synchronizing the global cache distribution topology to an edge node. And redundant storage and cloud load are effectively reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a real-time data shunting and intelligent caching method and system based on edge computing. BACKGROUND

[0002] With the development of technologies such as Internet of Things, artificial intelligence and 5G, the amount of data generated by terminal devices is growing explosively, and the traditional centralized computing model is facing a bottleneck in storage and computing resources. Edge computing, as a new computing mode, can effectively alleviate the pressure on the central server and improve data processing efficiency and real-time performance.

[0003] At present, the prior art scheme preliminarily processes terminal data through an edge node and caches part of the data. In some systems, preset rules are used to classify and shunt data, and some implement caching strategies based on data types or access frequencies, but still rely on static rules or a single indicator for decision-making, which cannot dynamically adapt to multi-dimensional data characteristics. The existing technologies generally have the following problems: first, there is a lack of more detailed structured processing and feature extraction of raw data, which makes it difficult to support fine-grained management; second, the data shunting strategy lacks intelligent support, making it difficult to balance real-time performance, security and cache hit rate; third, the edge cache update lacks a dynamic priority mechanism, which may cause high-value data to be mistakenly removed; fourth, the dynamic synchronization and collaborative update of the global cache topology have not been realized, and the overall efficiency and consistency of the system are low.

[0004] The present application provides a real-time data shunting and intelligent caching method and system based on edge computing. SUMMARY

[0005] The present application provides a real-time data shunting and intelligent caching method and system based on edge computing, which introduces a feature vector and a pre-trained shunting model to realize intelligent shunting of terminal data and dynamic cache priority management, effectively improving edge cache hit rate, data processing real-time performance, and overall system security and collaboration.

[0006] The present application provides a real-time data shunting and intelligent caching method based on edge computing, comprising: Step 1: collecting raw data streams of terminal devices and converting the raw data streams into structured data formats, calculating time efficiency weights, correlation weights and security levels based on the structured data to generate a feature vector; Step 2: outputting a first shunting signal and a second shunting signal through a pre-trained shunting decision model according to the feature vector; Step 3: extracting a feature identifier based on the first shunting signal and determining the corresponding edge cache partition of the feature identifier, obtaining the feature identifier and the corresponding historical access record, and determining the final priority score of the corresponding data in the edge cache. Step 4: managing edge cache space according to the priority score, removing low-priority data and generating cache update records; Step 5: based on the second shunt signal, performing security processing on the structured data, combining the cache update records and the transmission state of the security-processed data, generating a global cache distribution topology and synchronizing it to the edge node.

[0007] Preferably, based on the structured data, the time-sensitive weight, the correlation weight and the security level are calculated, including: Extracting the metadata field of the structured data format, obtaining the data collection timestamp and the data type identifier; Comparing the timestamp with the current system time to generate a time difference Δt; Matching the data type identifier with the preset data category coding table to output the standardized data type label; Based on the time difference Δt, performing time-sensitive evaluation logic to obtain the time-sensitive weight; Calling a preset data correlation graph engine, inputting the standardized data type label to the correlation graph query interface, and simultaneously retrieving a node set in the data correlation graph that has a direct dependency relationship with the corresponding data type, and counting the number of associated nodes; According to the preset rule, the correlation weight is calculated; According to the standardized data type label, triggering a multi-level matching process of the security policy library to determine the security level.

[0008] Preferably, a feature vector is generated, including: Reading the time-sensitive weight, the correlation weight and the security level from the buffer; Normalizing the time-sensitive weight, the correlation weight and the security level according to the preset vectorization rule; Assembling the normalized time-sensitive weight, the correlation weight and the security level into a feature vector in order.

[0009] Preferably, according to the feature vector, the first shunt signal and the second shunt signal are output by a pre-trained shunt decision model, including: Inputting the feature vector into the pre-trained shunt decision model, executing judgment logic combined with the first shunt condition and the second shunt condition to generate the first shunt signal and the second shunt signal.

[0010] Preferably, based on the first shunt condition and the second shunt condition, the judgment logic is executed to generate the first shunt signal and the second shunt signal, including: The first shunt condition is that if the time-sensitive weight is not less than the preset first time-sensitive threshold and the correlation weight is not less than the preset correlation threshold, it is determined as high-time-sensitive and high-correlation data, and the first shunt signal is generated. The second shunting condition is that if the security level is not less than a preset security level threshold or the time validity weight is less than a preset second time validity threshold, the data is determined as high-security or low-time-validity data, and a second shunting signal is generated; If the first shunting condition and the second shunting condition are not met, a first shunting signal is generated.

[0011] Preferably, a feature identifier is extracted based on the first shunting signal, and an edge cache partition corresponding to the feature identifier is determined, including: The first shunting signal is parsed, and a set of predefined key fields is extracted from the corresponding structured data; Hash operation is performed on the set of key fields to generate a globally unique feature identifier; The existence of the feature identifier in an edge cache meta-database is verified, if it exists, it is directly associated with the historical access record, if it does not exist, a new record is created in the meta-database and the access frequency is initialized; The verified feature identifier is assigned to an edge cache partition to determine a number of partitions.

[0012] Preferably, the feature identifier and the corresponding historical access record are obtained, and a final priority score of the corresponding data in the edge cache is determined, including: According to the feature identifier, historical access records are retrieved from a preset meta-database of the corresponding edge cache partition, and access frequency, recent access time, and access time delay sensitivity indicators are extracted; Based on a preset weight allocation strategy, a time decay coefficient is allocated to the access frequency, a time validity enhancement coefficient is allocated to the recent access time, and a service priority coefficient is allocated to the time delay sensitivity indicator, and a historical access comprehensive weight value is generated; The remaining storage space, computing load and network transmission delay data of the current edge cache partition are obtained, and the initial priority is determined in combination with the historical access comprehensive weight value; The final priority score is determined based on the historical access comprehensive weight value, the real-time resource state correction coefficient and the initial priority.

[0013] Preferably, the remaining storage space, computing load and network transmission delay data of the current edge cache partition are obtained, and the initial priority is determined in combination with the historical access comprehensive weight value, including: The remaining storage space, computing load and network transmission delay of the current edge cache partition are obtained, the storage resource utilization rate is determined based on the remaining storage space of the current edge cache partition, the computing load rate is determined based on the computing load of the current edge cache partition, the network delay rate is determined based on the network transmission delay of the current edge cache partition, and the storage resource utilization rate, the computing load rate and the network delay rate are normalized. An initial basic priority is determined based on the remaining storage space of the current edge cache partition after normalization processing, the computing load, and the network transmission delay data in combination with the historical access comprehensive weight value: wherein, is the initial basic priority, is the normalized historical access comprehensive weight value, ranging from , and is a preset weight coefficient, satisfying , is a resource state related factor, and the calculation formula is: wherein, ranges from , , and are weights corresponding to the storage resource utilization rate, the computing load rate, and the network delay rate of the current edge cache partition after normalization processing, and are dynamically updated based on an adaptive regulator, is the normalized storage resource utilization rate of the current edge cache partition, is the normalized computing load rate, is the normalized network delay rate; The initial basic priority obtained by calculation is truncated to obtain an initial priority: wherein, is the initial priority, is a preset upper priority threshold, is a preset lower priority threshold.

[0014] The real-time data shunting and intelligent caching system based on edge computing comprises: A data collection module collects original data streams of terminal devices and converts the original data streams into a structured data format, calculates time efficiency weight, correlation weight, and security level based on the structured data, and generates a feature vector; A signal shunting module outputs a first shunting signal and a second shunting signal through a pre-trained shunting decision model according to the feature vector; A data collection module extracts a feature identifier based on the first shunting signal, determines an edge cache partition corresponding to the feature identifier, acquires the feature identifier and corresponding historical access records, and determines a final priority score of the corresponding data in the edge cache; The cache generation module manages the edge cache space according to the priority score, removes low-priority data, and generates a cache update record; The cache synchronization module performs security processing on the structured data based on the second shunting signal, generates a global cache distribution topology in combination with the cache update record and the transmission state of the data after security processing, and synchronizes to the edge node.

[0015] Compared with the prior art, the application has the following beneficial effects: By introducing the feature vector and the pre-trained shunting model, intelligent shunting and dynamic cache priority management of terminal data are realized, and the edge cache hit rate, data processing real-time performance, and overall system security and collaboration are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is a flowchart of the real-time data shunting and intelligent cache method based on edge computing provided by the embodiments of the application.

[0018] Figure 2 is a structural diagram of the real-time data shunting and intelligent cache system based on edge computing provided by the embodiments of the application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the application clearer, the technical solutions in the application will be described clearly and completely in the following with reference to the drawings in the application. Obviously, the described embodiments are some embodiments of the application, not all embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0020] Embodiment 1: The embodiments of the application provide a real-time data shunting and intelligent cache method based on edge computing, as shown in Figure 1 , which comprises: Step 1: Collecting the original data stream of the terminal device, and converting the original data stream into a structured data format, calculating the time weight, correlation weight and security level based on the structured data, and generating a feature vector; Step 2: According to the feature vector, outputting a first shunting signal and a second shunting signal through a pre-trained shunting decision model; Step 3: Extract feature identifiers based on the first shunt signal, determine the corresponding edge cache partition, obtain the feature identifier and corresponding historical access record, and determine the final priority score of the corresponding data in the edge cache; Step 4: Manage edge cache space according to the priority score, remove low-priority data and generate cache update records; Step 5: Based on the second shunt signal, perform security processing on the structured data, combine the cache update records and the transmission state of the security-processed data, generate a global cache distribution topology and synchronize it to the edge node.

[0021] In this embodiment, the original data stream is the original data generated by the terminal device (such as a camera, a sensor), which may be a video clip, a temperature reading, a GPS coordinate, etc., and the format is chaotic (such as binary stream, JSON segment), for example: the real-time video stream of an intersection camera, generating one frame of image per second, with timestamp and location information.

[0022] In this embodiment, the structured data format is to organize the original data into a unified table or key-value pair for machine processing, for example: converting the video stream into a structured table, including fields: timestamp, camera ID, video frame data, scene type (such as traffic flow / pedestrian).

[0023] In this embodiment, the time weight is based on the freshness of the data. The higher the real-time performance, the higher the weight, for example: the real-time road condition data from an autonomous vehicle (time weight = 0.9) is more important than the historical data 10 minutes ago (weight = 0.2).

[0024] In this embodiment, the correlation weight is the degree of association between data and other data. The stronger the association, the higher the weight, for example: the heart rate data of a certain patient is associated with the blood oxygen data at the same time (weight = 0.8), which is more valuable than the isolated heart rate record (weight = 0.3).

[0025] In this embodiment, the security level is the sensitivity of the data. Private or critical data requires high security processing, for example: bank face recognition data (security level = high) needs to be encrypted, while weather sensor data (security level = low) can be transmitted in plaintext.

[0026] In this embodiment, the feature vector is a data tag (similar to the "barcode" of a commodity) that integrates the above weights, for example: the feature vector of a certain video frame = {time weight: 0.7, correlation weight: 0.5, security level: medium}.

[0027] In this embodiment, the shunting decision model is similar to a courier sorting system, which determines the destination of data based on the feature vector. The first shunting signal determines whether the data is cached to the edge node (a small server close to the terminal). For example, data with high time efficiency weight (such as 0.9) is sent to the edge cache (first signal = "cache"), and low time efficiency data (such as 0.2) is directly transmitted to the cloud. The second shunting signal determines the security processing method of the data. For example, high-security level data (such as medical records) triggers encryption (second signal = "AES-256 encryption"), and ordinary data is only compressed (second signal = "ZIP compression").

[0028] In this embodiment, the feature identifier is the unique identifier of the data (similar to the file name). For example, the video frame identifier = CAM001_20240615_120000.

[0029] In this embodiment, the historical access record is the frequency of past calls. For example, a certain identifier has been accessed 20 times in the past 1 hour (high frequency), and another identifier has been accessed only once (low frequency).

[0030] In this embodiment, removing low-priority data is like cleaning the cache on a mobile phone, deleting the lowest-rated files to free up space. For example, remove video frames with a score < 30, and keep high-priority data for real-time analysis.

[0031] In this embodiment, security processing performs operations (such as encryption, desensitization) according to the second signal. For example, after encrypting the patient's electrocardiogram data, it is transmitted to the edge node.

[0032] In this embodiment, the transmission status records whether the data is successfully cached or transmission fails. For example, identifier CAM001_20240615_120000 status = cached_edge_node_A.

[0033] In this embodiment, the global cache distribution topology is a data distribution map of all edge nodes. For example, the topology shows that node A stores real-time traffic data and node B stores medical data, which can be quickly queried and synchronized by other nodes.

[0034] The beneficial effects of the above technical solutions are: through edge computing, real-time shunting and intelligent caching optimize data processing, significantly reduce the delay, and improve the response speed. Based on time efficiency weight, correlation weight and security level, etc. Data characteristics, calculate the priority and dynamically manage the cache to ensure fast access to high-value data. The shunting decision model and the security processing mechanism work together to ensure data security while improving transmission efficiency. The global cache distribution topology realizes efficient data synchronization between edge nodes, enhances system collaboration, effectively reduces redundant storage and cloud load, and is suitable for intelligent transportation, industrial Internet of Things and other high-time efficiency scenarios, with the comprehensive advantages of high performance and low cost.

[0035] Embodiment 2: The embodiment of the present application provides a real-time data shunting and intelligent caching method based on edge computing, calculates time efficiency weight, correlation weight and security level based on structured data, and comprises the following steps of: extracting a metadata field in the structured data format, acquiring a data collection timestamp and a data type identifier; comparing the timestamp with a current system time to generate a time difference Δt; matching the data type identifier with a preset data category coding table to output a standardized data type mark; performing time efficiency evaluation logic based on the time difference Δt to obtain a time efficiency weight; calling a preset data correlation graph engine, inputting the standardized data type mark to a correlation graph query interface, simultaneously searching a node set having a direct dependency relationship with the corresponding data type in the data correlation graph, and counting the number of correlation nodes; calculating the correlation weight according to a preset rule; triggering a multi-stage matching process of a security policy library according to the standardized data type mark to determine a security level.

[0036] In the embodiment, the metadata field is extracted, and the metadata field is "description information" of data, which is similar to an attribute tag of a file. For example, in a piece of structured traffic data, the metadata includes a data collection timestamp: 2024-06-18 14:30:00 (the specific time of a camera shooting) and a data type identifier: Traffic_Video_Front (representing "front traffic video").

[0037] In the embodiment, the time difference Δt is calculated: the current time is compared to judge the new and old degree of data. For example, the current system time is 2024-06-18 14:35:00, which is 5 minutes different from the collection timestamp (Δt=5 minutes). In the embodiment, the time efficiency weight is obtained by performing time efficiency evaluation logic: based on the time difference Δt, the time efficiency evaluation logic is performed: if Δt≤a preset first time threshold, the time efficiency weight α0 is assigned a first priority coefficient; if Δt>the first time threshold but ≤a second time threshold, the time efficiency weight α0 is assigned a second priority coefficient; if Δt>the second time threshold, the time efficiency weight α0 is assigned a third priority coefficient; and the calculated time efficiency weight α0 is stored in a feature vector buffer.

[0038] In this embodiment, the correlation weight is calculated according to a preset rule: if the number of correlation nodes N is greater than or equal to a preset correlation node threshold, β0 is assigned a high correlation degree coefficient; if 0 < the number of correlation nodes N < the correlation node threshold, β0 is assigned a medium correlation degree coefficient; and if the number of correlation nodes N is 0, β0 is assigned a low correlation degree coefficient. The correlation weight β0 is written into the feature vector buffer.

[0039] In this embodiment, the standardized data type label is: a data category coding table: a pre-defined "data dictionary" with a unified name to avoid ambiguity. After the original identifier Traffic_Video_Front is matched with the coding table, the output standardized label is: traffic monitoring - front view - video stream. In this embodiment, according to the standardized data type label, a multi-level matching process of the security policy library is triggered to determine the security level: step A: searching the data type label in the security policy hash table, if hit, directly call its binding security level γ; step B: if not hit, traverse the identity certificate chain of the data source device to extract its security domain identifier; step C: according to the security domain identifier, match the default security level in the policy library and dynamically bind to the current data type; and push the finally determined security level γ to the feature vector buffer.

[0040] In this embodiment, the data correlation graph is similar to the relationship network of a social network, which shows the interaction relationship of different data types. The graph example is: the center node: traffic monitoring - front view - video stream, the correlation nodes: rear view video stream of the same intersection, radar vehicle speed data in the area, and red light status log.

[0041] In this embodiment, the number of correlation nodes is 3 (the more complex the dependency relationship is, the more the number is).

[0042] In this embodiment, the scene example is: an autonomous vehicle processing real-time road condition data, extracting metadata: timestamp: 2024-06-18 14:30:00, original identifier: CarX_LiDAR_Scan, calculating time difference: Δt = 2 minutes (the data is relatively new), standardized label: laser radar - point cloud - obstacle detection after matching the coding table, correlation graph query: finding that this data type is associated with camera vision data, high-precision map, and brake instruction record. The number of correlation nodes = 3 (indicating that this data affects multiple subsystems).

[0043] The beneficial effects of the above technical solutions are: through multi-dimensional feature extraction and weight calculation on structured data, combining data correlation graph and security policy library, fine evaluation of data value and security level division are realized, and the intelligence and accuracy of shunting and caching decision are improved.

[0044] Embodiment 3: The embodiment of the present application provides a real-time data shunting and intelligent caching method based on edge computing, generates a feature vector, and comprises the following steps: Reading the time limit weight, the correlation weight and the security level from the buffer; Normalizing the time limit weight, the correlation weight and the security level according to a preset vectorization rule; Assembling the normalized time limit weight, the correlation weight and the security level into a feature vector in sequence.

[0045] In the embodiment, the time limit weight, the correlation weight and the security level are normalized according to the preset vectorization rule, and before the feature vector is constructed, the system needs to perform standardization processing on the original weight indexes (the time limit weight, the correlation weight and the security level), so that they are unified to a specific numerical range, so as to avoid the influence of different dimensions on the final calculation. This process is called "normalization", and the rule is determined by the pre-set vectorization logic. For example, the time limit weight: if a certain data needs to be responded within 100ms, and another data is allowed to be delayed for 1s, after normalization, the time limit values of the two are adjusted to comparable values (such as 0.9 and 0.3) in the interval [0, 1], the correlation weight: the data of a sensor frequently interacts with other five devices, while another data is only used independently, and after normalization, the correlation degrees of the two are possibly mapped to 0.8 and 0.2, the security level: the security level of the data related to privacy is "high", and after normalization, it is converted to 0.9; the security level of the non-sensitive data is "low", and after normalization, it is converted to 0.1.

[0046] In the embodiment, the normalized time limit weight, the correlation weight and the security level are assembled into a feature vector in sequence, and the normalized weight indexes are arranged in a fixed order (such as time limit, correlation and security) to form a structured feature vector. The vector serves as a standardized representation of the data stream and is used for subsequent shunting decision model analysis. For example, after normalization, the time limit weight of a certain industrial equipment monitoring data is 0.7, the correlation weight is 0.5, and the security level is 0.3, and the feature vector [0.7, 0.5, 0.3] is assembled in sequence. The emergency braking signal of an autonomous vehicle may generate a feature vector [0.9, 0.6, 0.8], reflecting the high time limit, medium correlation and high security requirements.

[0047] The beneficial effects of the above technical solutions are as follows: by normalizing the multi-dimensional weight indexes and constructing a feature vector, the standardized expression of data features is realized, a unified input basis is provided for subsequent shunting models, and the accuracy and system compatibility of data shunting decision are improved.

[0048] Embodiment 4: The embodiment of the present application provides a real-time data shunting and intelligent caching method based on edge computing, and outputs a first shunting signal and a second shunting signal through a pre-trained shunting decision model according to the feature vector, comprising the following steps: The feature vector is input into a pre-trained shunting decision model to execute judgment logic in combination with the first shunting condition and the second shunting condition, to generate the first shunting signal and the second shunting signal.

[0049] In this embodiment, the pre-trained shunting decision model is an intelligent decision module based on machine learning or deep learning, which has been trained and optimized before system deployment and can automatically determine the processing mode of the data stream according to the input feature vector. The model learns the optimal balance between priority, security and efficiency by analyzing the shunting effect of historical data, so as to intelligently output the shunting signal in actual operation without real-time parameter adjustment. The pre-trained shunting decision model has completed data sample training in the offline stage and has the ability to dynamically determine the data flow direction based on the feature vector. In runtime, the model directly loads the weight parameters and combines the preset shunting conditions for efficient inference to generate a shunting decision signal, ensuring low latency and high accuracy. For example, in an industrial Internet of Things scenario, if the feature vector indicates that the device data has high timeliness (e.g., 0.9) but low security (e.g., 0.2), the model may output the first shunting signal (preferentially sent to a near-end edge node), while high data correlation (e.g., 0.7) triggers the second shunting signal (synchronized to a cooperative edge cluster). In an intelligent transportation scenario, the feature vector of an emergency vehicle with high timeliness (0.95) and high security (0.8) may trigger the first shunting signal (directly to a cloud emergency platform), while ordinary traffic flow data is processed in batches through the second shunting signal (cached in an edge node).

[0050] The above technical solution has the following beneficial effects: by introducing a pre-trained shunting decision model and combining multiple shunting conditions, intelligent shunting judgment of data is realized, the self-adaptive ability and processing efficiency of the shunting strategy are improved, and real-time decision-making requirements in complex scenarios are met.

[0051] Embodiment 5: The embodiment of the application provides a real-time data shunting and intelligent caching method based on edge computing, executes judgment logic based on a first shunting condition and a second shunting condition, and generates a first shunting signal and a second shunting signal, which comprises the following steps: The first shunting condition is that if the timeliness weight is not less than a preset first timeliness threshold and the correlation weight is not less than a preset correlation threshold, the data is determined as high-timeliness high-correlation data, and the first shunting signal is generated; The second shunting condition is that if the security level is not less than a preset security level threshold or the timeliness weight is less than a preset second timeliness threshold, the data is determined as high-security or low-timeliness data, and the second shunting signal is generated; If the first shunting condition and the second shunting condition are not met, the first shunting signal is generated.

[0052] In this embodiment, dynamic modification of the shunting signal is also included: querying the real-time load status of the edge node, if the remaining space of the edge cache is lower than the dynamic threshold, then the first shunting signal is appended with a downgrade instruction; combining the historical access trend of the data feature identifier, the cloud storage strategy in the second shunting signal is supplemented with a prefetch rule.

[0053] In this embodiment, the preset first timeliness threshold is a pre-set timeliness determination reference value, which is used to distinguish the sensitivity of data to real-time. If the timeliness weight of the data is not lower than the threshold, it indicates that it has strict restrictions on transmission or processing delay. For example, if the first timeliness threshold is set to 0.7, if the timeliness weight of the vibration data of an industrial equipment is 0.8 (response within 50 ms is required), it is determined as high timeliness data; if the timeliness weight of another environmental temperature and humidity data is 0.5 (500 ms delay is allowed), it does not reach the threshold.

[0054] In this embodiment, the preset correlation threshold is a boundary value for measuring the coordination needs of data with other devices or services. If the correlation weight is not lower than the threshold, it indicates that the processing of the data needs to rely on multi-party coordination. For example, if the correlation threshold is set to 0.6, in a certain intelligent factory, the correlation weight of the mechanical arm control instruction is 0.9 (needs to be linked with multiple sensors), which meets the threshold; and the correlation weight of a single temperature sensor data is 0.3 (independent reporting), which does not meet the condition.

[0055] In this embodiment, the preset security level threshold is a security sensitivity dividing value. If the security level of the data is not lower than this value, it indicates that it involves privacy or critical systems and needs special protection. For example, if the security level threshold is set to 0.8, the security level of patient medical image data is 0.9 (needs to be transmitted in encrypted form), which triggers high security determination; the security level of ordinary advertising push data is 0.2, which does not need special processing.

[0056] In this embodiment, the preset second timeliness threshold is a secondary determination reference of timeliness. Data below this value is considered to have low timeliness requirements and can tolerate higher delay or batch processing. For example, if the second timeliness threshold is set to 0.3, the timeliness weight of historical log data is 0.2 (allowing minute-level delay), which is determined as low timeliness; and the timeliness weight of real-time video stream is 0.6, which is still higher than the threshold.

[0057] The beneficial effects of the above technical solutions are: by setting multi-dimensional shunting conditions and combining threshold judgment mechanism, accurate identification and classification of different value and security attribute data are realized to generate shunting signals, which improves the pertinence of data scheduling and the efficiency of system resource utilization.

[0058] Embodiment 6: The embodiment of the present application provides a real-time data shunting and intelligent caching method based on edge computing, feature identifiers are extracted based on the first shunting signal, and an edge cache partition corresponding to the feature identifiers is determined, and the method comprises the following steps: The first shunting signal is parsed, and a set of predefined key fields is extracted from the corresponding structured data; Hash operation is performed on the set of key fields to generate globally unique feature identifiers; The existence of the feature identifiers in the edge cache meta database is verified, if the feature identifiers exist, the feature identifiers are directly associated with historical access records, if the feature identifiers do not exist, a new record is created in the meta database and the access frequency is initialized; The verified feature identifiers are allocated to edge cache partitions to determine a plurality of partitions.

[0059] In the embodiment, the set of key fields comprises a data source identifier, a business type code and a timestamp; In the embodiment, the set of key fields is processed by a hash function to generate unique feature identifiers, so that the identifiers corresponding to the key field combinations of different data sources have global unique identification, and hash conflicts in the edge cache partition are avoided, and an example is as follows: in the Internet of Vehicles application, the set of key fields can comprise a vehicle ID, a sensor type and a timestamp, and after hash operation, a unique feature identifier (such as A1B2-C3D4-E5F6) is generated. In the industrial Internet of Things scene, if the key fields are a device serial number, a data acquisition period and a data type, the hash operation can generate a unique identifier (such as X9Y8-Z7W6-V5U4) for subsequent edge cache management.

[0060] In the embodiment, the existence of the feature identifiers in the edge cache meta database is verified: by querying the edge cache meta database, it is determined whether the feature identifiers are stored in the cache index, if the feature identifiers exist, the feature identifiers are associated with the existing cache record, if the feature identifiers do not exist, a new meta data entry is initialized and an initial frequency parameter is set, and an example is as follows: the existence case: if the feature identifier (such as K2J3-H4G5-F6D7) of a smart meter exists in the meta database record, the past cache data (such as the last 3 readings) of the meter is directly associated, and the access frequency is updated; the nonexistence case: if the feature identifier (such as N8M7-B6V5-C4X3) of a newly connected unmanned aerial vehicle does not exist in the meta database, a new record is created for the identifier, the access frequency is initialized to 1, and the edge cache partition is allocated.

[0061] In this embodiment, metadata verification and partition allocation: existing → direct association history record, update access frequency (such as from 5 to 6); non-existent → new metadata record, initialize frequency to 1, and assign to appropriate edge partition (such as high-frequency access data to low-latency partition).

[0062] In this embodiment, edge cache partitioning is performed: the hash value of the feature identifier is calculated, the candidate cache partition is determined based on the consistent hashing algorithm; the real-time load state of the candidate partition is queried, and if the current storage pressure exceeds the dynamic threshold, the suboptimal partition is selected according to the proximity principle.

[0063] The beneficial effects of the above technical solutions are: by extracting key fields and generating unique feature identifiers, precise association of data and edge cache and efficient partition management are realized, and the utilization rate of cache resources and the response speed of data access are effectively improved.

[0064] Embodiment 7: The embodiment of the application provides a real-time data shunting and intelligent caching method based on edge computing, acquires the feature identifier and the corresponding historical access record, and determines the final priority score of the corresponding data in the edge cache, including: According to the feature identifier, historical access records are retrieved from the preset metadata database of the corresponding edge cache partition, and access frequency, recent access time and access time delay sensitivity indicators are extracted; Based on a preset weight allocation strategy, a time decay coefficient is allocated to the access frequency, a time effectiveness enhancement coefficient is allocated to the recent access time, and a service priority coefficient is allocated to the time delay sensitivity indicator, and a historical access comprehensive weight value is generated; Get the remaining storage space, computing load and network transmission delay data of the current edge cache partition, combine the historical access comprehensive weight value, and determine the initial priority; Determine the final priority score based on the historical access comprehensive weight value, the real-time resource state correction coefficient and the initial priority.

[0065] In this embodiment, according to the feature identifier, the historical access record is retrieved from the preset metadata database of the corresponding edge cache partition, the access frequency, the latest access time and the access time delay sensitivity index are extracted, including: the access frequency (such as 10 times / hour): indicating the number of calls of the data in a certain time period, the latest access time (such as 2024-03-2014:30:00): recording the time point when the data was last requested, the access time delay sensitivity index (such as 0.8): quantifying the tolerance of the data to processing delay (the higher the value, the more stringent the timeliness requirement), example: industrial Internet of Things scene: the feature identifier of certain machine tool vibration data (such as V2B3-N4M5-K6L7) corresponds to the metadata database, which records its access frequency = 50 times / minute (high frequency), latest access time = current time-5 seconds (active), time delay sensitivity = 0.9 (need millisecond level response); smart home scene: the feature identifier of air conditioner room temperature data (such as H7J8-K9L0-P1Q2) may show access frequency = 2 times / hour, latest access time = 3 hours ago, time delay sensitivity = 0.3 (allowing second level delay).

[0066] In this embodiment, based on the preset weight allocation strategy, the time decay coefficient is allocated to the access frequency, the timeliness enhancement coefficient is allocated to the latest access time, and the service priority coefficient is allocated to the time delay sensitivity index, and the historical access comprehensive weight value is generated: the time decay coefficient is given to the access frequency: reducing the long-term impact of historical access records (such as half-life model), ensuring that high-frequency but outdated data will not occupy high priority for a long time, the timeliness enhancement coefficient is given to the latest access time: enhancing the weight of recent active data (such as exponential time addition), reflecting its current popularity and prediction value, the service priority coefficient is given to the time delay sensitivity: adjusting its contribution proportion to the final score according to the business type (such as autonomous driving vs. log backup), and finally the generated historical access comprehensive weight value integrates the above coefficients to form a dynamic priority basis, example: vehicle networking emergency data: access frequency 100 times / minute × time decay coefficient 0.9 (high frequency but need to prevent historical data interference), latest access time current-0.1 second × timeliness enhancement coefficient 1.2 (extremely high timeliness addition), time delay sensitivity 1.0 (zero tolerance delay) × service priority coefficient 1.5 (safety critical business), comprehensive weight value: 100 × 0.9 + 1.2 × Max (recentness) + 1.0 × 1.5 = priority score 92.7 (extremely high), retail inventory data: access frequency 5 times / hour × time decay coefficient 0.6, latest access time current-2 hours × timeliness enhancement coefficient 0.8, time delay sensitivity 0.4 × service priority coefficient 1.0, comprehensive weight value: 5 × 0.6 + 0.8 × medium level timeliness + 0.4 × 1.0 = priority score 4.2 (lower).

[0067] In this embodiment, the final priority score is determined based on the historical access comprehensive weight value, the real-time resource state correction coefficient and the initial priority: wherein, is the final priority score, is the initial priority weight, which is dynamically attenuated from 0.6 to the lowest 0.4 according to the resource tension degree, and the real-time responsiveness is improved, is the initial priority, such as the initial score of 0.8 of the automatic driving camera data in the resource good state, is the historical weight factor, β = 1 - α, and the influence control of the historical item is enhanced when the resource is tense, is the resource suppression coefficient, is the historical access comprehensive weight value, is the real-time resource state correction coefficient, which dynamically quantifies the node resource tension degree, and the greater the value, the more tense the resource is, and if the node storage is tense, the CPU is highly occupied and the delay is high, which can reach 2.5; wherein, : the current remaining storage space proportion of the edge node; : the ratio of the current CPU utilization rate to the maximum capacity; : the ratio of the current network transmission delay to the maximum delay allowed by the business; , , are the weight coefficients of the storage, calculation and network delay dimensions, respectively, satisfying the normalization constraint : the default configuration is: = 0.4, = 0.3, = 0.3.

[0068] The beneficial effects of the above technical solution are: by combining the historical access features and the real-time resource state to score the priority of the cache data, dynamic and accurate cache management and space scheduling are realized, and the data hit rate and system processing efficiency in the edge computing environment are improved.

[0069] Embodiment 8: The embodiment of the application provides a real-time data shunting and intelligent caching method based on edge computing, acquires the residual storage space, calculation load and network transmission delay data of the current edge cache partition, combines the historical access comprehensive weight value, determines the initial priority, and comprises the following steps: ​​obtain the remaining storage space, the computing load and the network transmission delay of the current edge cache partition, determine the storage resource utilization rate based on the remaining storage space of the current edge cache partition, determine the computing load rate based on the computing load of the current edge cache partition, determine the network delay rate based on the network transmission delay of the current edge cache partition, and perform normalization processing on the storage resource utilization rate, the computing load rate and the network delay rate; determine the initial basic priority based on the normalized remaining storage space, the normalized computing load and the normalized network transmission delay of the current edge cache partition in combination with the historical access comprehensive weight value: wherein, is the initial basic priority, is the normalized historical access comprehensive weight value, and the range is , and is a preset weight coefficient, and satisfies , is a resource state related factor, and the calculation formula is: wherein, the range is , , and are weights corresponding to the normalized storage resource utilization rate, the normalized computing load rate and the normalized network delay rate of the current edge cache partition respectively, and are dynamically updated based on an adaptive regulator, is the normalized storage resource utilization rate of the current edge cache partition, is the normalized computing load rate, is the normalized network delay rate. perform truncation processing on the initial basic priority to obtain an initial priority: wherein, is the initial priority, is a preset upper priority threshold, is a preset lower priority threshold.

[0070] In this embodiment, the remaining storage space of the current edge cache partition is obtained : a storage management interface of the edge node operating system, periodically polls the disk block allocation state of the target cache partition, and multiplies the number of unallocated physical blocks by the block size to obtain the remaining storage space capacity In Linux systems, the number of remaining blocks in a partition can be obtained using the statvfs() or df command, and then calculated according to the formula: In Windows systems, the remaining number of bytes can be read using GetDiskFreeSpaceEx(). For distributed storage middleware interfaces, if using in-memory databases such as Redis, the INFOMEMORY command can be called to obtain used_memory and maxmemory.

[0071] In this embodiment, the computing load By analyzing process scheduling statistics provided by the operating system kernel, the proportion of time the CPU cores are in a non-idle state per unit time is calculated, and a weighted average of multi-core load is aggregated: In Windows systems, GetSystemTimes() is used to obtain kernel and user-space time. For containerized environments: if running in a Docker / K8s environment, container-level CPU usage can be obtained through the cgroups cpuacct subsystem.

[0072] In this embodiment, the network transmission delay is determined by periodically sending lightweight probe messages to the data requester, calculating the time difference between the message transmission and the received response, removing outliers, and taking the 90th percentile within the sliding window as the effective delay value.

[0073] In this embodiment, the storage resource utilization rate is determined based on the remaining storage space of the current edge cache partition. : That is: first calculate the ratio of remaining storage space to total storage space; subtract the ratio from 1 to get the percentage of used space; finally multiply by 100% to convert it to a percentage. In this embodiment, the computing load rate is determined based on the computing load of the current edge cache partition. This measure comprehensively assesses the load on computing resources by combining CPU utilization and memory utilization. The calculation formula is as follows: in, It is a configurable weighting factor used to control the contribution ratio of CPU and memory to the computing load. It can be adjusted according to business needs. This is to assess the utilization of CPU or memory resources.

[0074] In this embodiment, the network latency rate is determined based on the network transmission latency of the current edge cache partition. The network latency rate is used to measure the deviation of the current network latency from the baseline latency and the maximum allowed latency, and the calculation formula is as follows: The formula reflects the difference between the current transmission latency and the predetermined quality of service requirement. The larger the value, the greater the network latency and the poorer the performance.

[0075] In this embodiment, the adaptive adjuster is used to dynamically update the priority as follows: The updated formula is as follows: Reflects the impact of storage resource usage on priority, is the base weight, representing the initial weight without any adjustment, is the activation threshold of storage space, used to control the impact of storage space usage on weight, is the first adjustment intensity coefficient, determining the impact of storage space usage on weight coefficient, is the hyperbolic tangent function, with output value range between -1 and 1, which makes the weight adjustment change less when the storage space utilization rate is close to the activation threshold, and produces larger adjustment when it fluctuates greatly.

[0076] The updated formula is as follows: Reflects the impact of computing resource load on priority, is the normalized computing load utilization rate, is the activation threshold of computing load, used to control the impact of computing load usage on weight, is the sigmoid function, with output value range between 0 and 1. It makes the weight close to 1 when the computing load is low, and close to 0 when the load is high. That is, the higher the computing load, the greater the negative impact on priority, is the second adjustment intensity coefficient, determining the impact of computing load on weight; The updated formula is as follows: Reflects the impact of network latency on priority, is the normalized network latency rate (range between 0 and 1), representing the ratio of the current network latency to the maximum allowed latency, is the third adjustment intensity coefficient, determining the impact of network latency on weight. This formula indicates that when the network latency increases, The value will also increase, thus affecting the priority calculation and preventing excessive delay from causing the priority to be too low.

[0077] The beneficial effects of the above technical solution are as follows: by introducing resource status-related factors and adaptive adjustment mechanisms, comprehensively considering the dynamic changes in cache space, computing load and network latency, accurately calculating the initial priority of data, and improving the real-time performance of cache management and the efficiency of system resource utilization.

[0078] Example 9: This invention provides a real-time data offloading and intelligent caching system based on edge computing, such as... Figure 2 As shown, it includes: Data acquisition module: Acquires raw data streams from terminal devices, converts the raw data streams into structured data format, calculates timeliness weight, correlation weight, and security level based on the structured data, and generates feature vectors; Signal splitting module: Based on the feature vector, it outputs a first split signal and a second split signal through a pre-trained splitting decision model; The scoring determination module extracts feature identifiers based on the first diversion signal, determines the edge cache partition corresponding to the feature identifier, obtains the feature identifier and the corresponding historical access record, and determines the final priority score of the corresponding data in the edge cache. Cache generation module: manages edge cache space according to the priority score, removes low-priority data and generates cache update records; Cache synchronization module: Based on the second diversion signal, it performs security processing on the structured data, and combines the cache update record and the transmission status of the data after security processing to generate a global cache distribution topology and synchronize it to the edge nodes.

[0079] The beneficial effects of the above technical solution are as follows: Real-time data offloading and intelligent caching through edge computing significantly reduce latency and improve response speed. Based on data characteristics such as timeliness weight, correlation weight, and security level, priorities are calculated and cache is dynamically managed to ensure rapid access to high-value data. The offloading decision model and security processing mechanism work together to improve transmission efficiency while ensuring data security. The global cache distribution topology enables efficient data synchronization between edge nodes, enhances system synergy, effectively reduces redundant storage and cloud load, and is suitable for high-timeliness scenarios such as intelligent transportation and industrial IoT, combining the advantages of high performance and low cost.

[0080] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A real-time data offloading and intelligent caching method based on edge computing, characterized in that, include: Step 1: Collect the raw data stream from the terminal device and convert the raw data stream into a structured data format. Calculate the timeliness weight, correlation weight, and security level based on the structured data to generate a feature vector. Step 2: Based on the feature vector, output the first diversion signal and the second diversion signal through the pre-trained diversion decision model; Step 3: Extract feature identifiers based on the first diversion signal, determine the edge cache partition corresponding to the feature identifier, obtain the feature identifier and the corresponding historical access record, and determine the final priority score of the corresponding data in the edge cache; Step 4: Manage the edge cache space according to the priority score, remove low-priority data and generate cache update records; Step 5: Based on the second diversion signal, perform security processing on the structured data, and combine the cache update record and the transmission status of the data after security processing to generate a global cache distribution topology and synchronize it to the edge nodes.

2. The real-time data offloading and intelligent caching method based on edge computing according to claim 1, characterized in that, Based on the structured data, the timeliness weight, correlation weight, and security level are calculated, including: Extract the metadata fields of the structured data format to obtain the data collection timestamp and data type identifier; The timestamp is compared with the current system time to generate a time difference Δt; The data type identifier is matched with a preset data category encoding table to output a standardized data type tag; Based on the time difference Δt, the timeliness assessment logic is executed to obtain the timeliness weight; Call the preset data association graph engine, input the standardized data type tag to the association graph query interface, and at the same time retrieve the set of nodes in the data association graph that have a direct dependency relationship with the corresponding data type, and count the number of associated nodes; Calculate the association weights according to preset rules; Based on the standardized data type tags, a multi-level matching process of the security policy library is triggered to determine the security level.

3. The real-time data offloading and intelligent caching method based on edge computing according to claim 1, characterized in that, Generate feature vectors, including: Read the timeliness weight, association weight, and security level from the buffer; The timeliness weight, association weight, and security level are normalized according to the preset vectorization rules; The normalized timeliness weight, correlation weight, and security level are assembled into a feature vector in sequence.

4. The real-time data offloading and intelligent caching method based on edge computing according to claim 1, characterized in that, Based on the feature vector, a pre-trained diversion decision model outputs a first diversion signal and a second diversion signal, including: The feature vector is input into the pre-trained diversion decision model, and the judgment logic is executed in combination with the first diversion condition and the second diversion condition to generate the first diversion signal and the second diversion signal.

5. The real-time data offloading and intelligent caching method based on edge computing according to claim 4, characterized in that, Based on the first and second shunt conditions, a judgment logic is executed to generate the first shunt signal and the second shunt signal, including: The first diversion condition is that if the timeliness weight is not less than the preset first timeliness threshold and the correlation weight is not less than the preset correlation threshold, then it is determined to be high timeliness and high correlation data, and the first diversion signal is generated. The second diversion condition is that if the security level is not less than the preset security level threshold or the timeliness weight is less than the preset second timeliness threshold, then it is determined to be high security or low timeliness data, and a second diversion signal is generated. If the first shunt condition and the second shunt condition are not met, then the first shunt signal is generated.

6. The real-time data offloading and intelligent caching method based on edge computing according to claim 1, characterized in that, Extracting feature identifiers based on the first shunt signal and determining the edge buffer partitions corresponding to the feature identifiers includes: Analyze the first split signal and extract a predefined set of key fields from the corresponding structured data; Perform a hash operation on the key field set to generate a globally unique feature identifier; Verify the existence of the feature identifier in the edge cache metadata database. If it exists, it is directly associated with the historical access record. If it does not exist, a new record is created in the metadata database and the access frequency is initialized. The verified feature identifiers are allocated to edge cache partitions to determine several partitions.

7. The real-time data offloading and intelligent caching method based on edge computing according to claim 1, characterized in that, Obtain the feature identifier and the corresponding historical access record, and determine the final priority score of the corresponding data in the edge cache, including: Based on the feature identifier, historical access records are retrieved from the preset metadata database of the corresponding edge cache partition, and access frequency, recent access time, and access latency sensitivity index are extracted. Based on a preset weight allocation strategy, a time decay coefficient is assigned to the access frequency, a timeliness enhancement coefficient is assigned to the most recent access time, and a business priority coefficient is assigned to the latency sensitivity index to generate a comprehensive weight value for historical access. Obtain the remaining storage space, computing load, and network transmission latency data of the current edge cache partition, and determine the initial priority by combining the historical access comprehensive weight value; The final priority score is determined based on the historical access weight value, the real-time resource status correction coefficient, and the initial priority.

8. The real-time data offloading and intelligent caching method based on edge computing according to claim 7, characterized in that, Obtain the remaining storage space, computational load, and network transmission latency data of the current edge cache partition. Combine this with the historical access weight value to determine the initial priority, including: The remaining storage space, computing load, and network transmission latency of the current edge cache partition are obtained. The storage resource utilization rate is determined based on the remaining storage space of the current edge cache partition, the computing load rate is determined based on the computing load of the current edge cache partition, and the network latency rate is determined based on the network transmission latency of the current edge cache partition. The storage resource utilization rate, computing load rate, and network latency rate are then normalized. Based on the normalized remaining storage space, computational load, and network transmission latency data of the current edge cache partition, combined with the historical access weight value, the initial basic priority is determined: in, As the initial base priority, This is the normalized historical access weight value, ranging from [value missing]. , and The preset weighting coefficients satisfy... , The resource status-related factor is calculated using the following formula: in, The range is , , and These are the weights corresponding to the storage resource utilization, computing load, and network latency of the current edge cache partition after normalization, and are dynamically updated based on an adaptive adjuster. This represents the normalized storage resource utilization rate of the current edge cache partition. The normalized computational load rate. This represents the network latency rate after normalization. The calculated initial base priority is truncated to obtain the initial priority: in, As the initial priority, To preset the upper limit threshold, This is the preset priority lower limit threshold.

9. A real-time data offloading and intelligent caching system based on edge computing, characterized in that, include: Data acquisition module: Acquires raw data streams from terminal devices, converts the raw data streams into structured data format, calculates timeliness weight, correlation weight, and security level based on the structured data, and generates feature vectors; Signal splitting module: Based on the feature vector, it outputs a first split signal and a second split signal through a pre-trained splitting decision model; Data acquisition module: Extracts feature identifiers based on the first diversion signal, determines the edge cache partition corresponding to the feature identifier, obtains the feature identifier and the corresponding historical access record, and determines the final priority score of the corresponding data in the edge cache; Cache generation module: manages edge cache space according to the priority score, removes low-priority data and generates cache update records; Cache synchronization module: Based on the second diversion signal, it performs security processing on the structured data, and combines the cache update record and the transmission status of the data after security processing to generate a global cache distribution topology and synchronize it to the edge nodes.

Images