Intelligent cache processing method for real-time labeling data stream

Abstract

The application discloses a real-time intelligent cache processing method for labeled data stream, and belongs to the technical field of cross-domain data processing, comprising the following steps: receiving a multi-source labeled data stream, mapping fields and calculating similarity through a time sequence graph neural network, generating a mapping rule, performing incremental cleaning, and outputting a structured labeled data stream; extracting data entities, attributes and relationships, constructing a heterogeneous cache graph and establishing a four-element composite index; then constructing a hot-warm-cold layered cache architecture, dynamically distributing data entities based on real-time heat entropy, relationship strength and the like, synchronously updating the graph and the index; identifying low-confidence data through entity confidence evaluation, generating a correction suggestion, feeding back cleaning, and synchronously updating the parameters of each data source subgraph copy and the state of hot entities with the aid of a security aggregation algorithm; and realizing fine-grained dynamic access control based on a deep policy evaluation network; the method improves the cache efficiency, data reliability and access security of multi-source labeled data, and is suitable for real-time data stream processing scenarios.

Description

Technical Field

[0001] This invention belongs to the field of cross-domain data processing technology, specifically a method for intelligent caching of real-time labeled data streams. Background Technology

[0002] With the widespread application of data annotation technology in fields such as artificial intelligence and big data analytics, real-time annotated data streams exhibit multi-source heterogeneity and dynamic high-frequency characteristics, posing severe challenges to data processing and cache management. Existing technologies lack precise semantic and structural mapping mechanisms for multi-source annotated fields, easily leading to issues such as field conflicts and inconsistent formats, resulting in low data structuring efficiency and difficulty in meeting real-time processing requirements. Traditional caching architectures often employ single-level or static allocation strategies, failing to fully consider the relationships between data entities and real-time changes in popularity, resulting in wasted cache resources and high access response latency. Simultaneously, existing solutions lack dynamic evaluation and correction mechanisms for the confidence level of annotated data, allowing low-confidence data to easily flow into subsequent processes, affecting the reliability of data applications. Furthermore, in distributed data source scenarios, there are privacy risks during data synchronization, and access control is mostly coarse-grained, unable to dynamically adjust permissions based on data attributes, subject identity, etc., making it difficult to balance data security and access flexibility. These problems collectively restrict the processing efficiency, data quality, and security compliance of real-time annotated data streams, urgently requiring an efficient and intelligent caching solution. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention proposes an intelligent caching method for real-time labeled data streams. It receives multi-source labeled data streams, maps fields using a time-series graph neural network and calculates similarity, generates mapping rules for incremental cleaning, and outputs a structured labeled data stream. Data entities, attributes, and relationships are extracted to construct a heterogeneous cache graph and establish a four-element composite index. Subsequently, a hot-warm-cold hierarchical caching architecture is constructed, dynamically allocating data entities based on real-time heat entropy and relationship strength, and synchronously updating the graph and index. Low-confidence data is identified through entity confidence assessment, and correction suggestions are generated for feedback cleaning. A secure aggregation algorithm synchronizes the subgraph replica parameters and hot entity status of each data source. Fine-grained dynamic access control is achieved based on a deep policy evaluation network. This method improves the caching efficiency, data reliability, and access security of multi-source labeled data, and is suitable for real-time data stream processing scenarios.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] Intelligent caching methods for real-time labeled data streams include:

[0006] S1: Receives multi-source labeled data streams, maps labeled fields from different sources to nodes of a temporal graph neural network, calculates semantic and structural similarity between nodes, generates field mapping rules and performs incremental cleaning, and outputs structured labeled data streams.

[0007] S2: Using the structured labeled data stream as input, extract the data entities, entity attributes, and relationships between entities contained therein, construct a heterogeneous cache graph, and establish a multi-dimensional index;

[0008] S3: Based on the heterogeneous cache graph, construct a hierarchical cache architecture that includes a hot layer, a warm layer and a cold layer, dynamically allocate data entities in the heterogeneous cache graph to the corresponding cache layer, and synchronously update the heterogeneous cache graph status and multi-dimensional index in real time.

[0009] S4: Based on the association and confidence characteristics of each data entity in the heterogeneous cache graph, identify the labeled data, generate data correction suggestions and feed them back to the incremental cleaning process to update the structured labeled data stream and the heterogeneous cache graph. At the same time, maintain the corresponding subgraph copy of the heterogeneous cache graph locally on each data source node, and synchronize the graph neural network model parameters and hot spot entity status of each subgraph copy through a secure aggregation algorithm.

[0010] S5: Based on the attributes and association characteristics of data entities in the heterogeneous cache graph, perform dynamic access control on the data stored in the hierarchical cache architecture, and feed the control decision back to S3.

[0011] Specifically, S1 includes:

[0012] S1.1: Create a temporal feature node for each labeled field from each source; the attributes of the temporal feature node include the semantic embedding vector of the field name, the statistical feature vector of the value range distribution, and the timestamp sequence of field occurrence and change events;

[0013] S1.2: Input all temporal feature nodes within the continuous time window into the temporal graph neural network. The temporal graph neural network captures the temporal dependency features of each temporal feature node through the time-gated recurrent unit layer, and then aggregates the cross-source structural features between temporal feature nodes from different data sources through the graph attention layer, and outputs the fused feature vector of each temporal feature node.

[0014] S1.3: Calculate the cross-source semantic affinity between the fused feature vectors of any two temporal feature nodes, and calculate the temporal co-occurrence intensity based on the co-occurrence relationship of the two temporal feature nodes within the sliding time window. Then, input the cross-source semantic affinity and the temporal co-occurrence intensity into the rule generation neural network. The rule generation neural network outputs a probabilistic field mapping rule matrix, where each element in the field mapping rule matrix represents the confidence weight of mapping one source field to another source field.

[0015] S1.4: Based on the field mapping rule matrix, the original labeled fields that flow in in real time are parsed and aligned. Data with a mapping confidence level higher than the preset first threshold are directly fused. Data with a confidence level lower than the preset first threshold but higher than the preset second threshold are marked and temporarily stored in the pending area. Data with a confidence level lower than the preset second threshold is triggered with an anomaly alarm. Finally, a structured labeled data stream with uniform fields and standardized format is output.

[0016] Specifically, the process of constructing the heterogeneous cache map in S2 includes:

[0017] S2.1: Identify data entities from the structured labeled data stream and assign a globally unique entity fingerprint code to each data entity; the entity fingerprint code is generated by combining its entity type hash value and attribute hash value;

[0018] S2.2: Extract the attribute key-value pairs of each data entity, instantiate each attribute key-value pair into an attribute node, and link each attribute node to its data entity through an edge;

[0019] S2.3: Identify the relationships between different data entities, instantiate each relationship as a directed relationship edge, and assign the relationship edge a relationship type, establishment time, and a relationship strength value based on co-occurrence frequency and semantic consistency;

[0020] S2.4: Construct a heterogeneous cache graph based on all identified data entities, all generated attribute nodes and their links to data entities, and all generated relational links. The heterogeneous cache graph is composed of data entities as vertices, attribute nodes as subordinate vertices, and links between data entities and attribute nodes and relational links between data entities as edges.

[0021] Specifically, the multi-dimensional index in S2 is a four-element composite index structure, which includes an attribute inverted index, a relation adjacency index, a time-series heat index, and a subgraph structure fingerprint index.

[0022] The attribute inverted index is an inverted list built for the keys and values ​​of all attribute nodes, used for fuzzy matching and range queries based on attribute values;

[0023] The relation adjacency index is used to record the relation type, target data entity, and current relation strength value of all relation connection edges for each data entity, and is sorted in descending order of relation strength value;

[0024] The time-series heat index is based on the event timestamps of when a data entity is accessed, associated, or its attributes are modified. It uses a time decay function to dynamically calculate the real-time heat entropy of each data entity and establishes a sorted index of the real-time heat entropy.

[0025] The subgraph structure fingerprint index calculates the local subgraph structure within two hops centered on each data entity, generating a fixed-length subgraph structure fingerprint for graph-based similarity retrieval.

[0026] Specifically, the steps of S3 include:

[0027] S3.1: Divide the cache storage medium into three logical layers, including a hot layer, a warm layer, and a cold layer;

[0028] S3.2: Based on the heterogeneous cache map, a cache location decision function dynamically allocates data entities in the heterogeneous cache map to the divided cache layers; the cache location decision function uses the real-time heat entropy of the target data entity, the average relationship strength value of its associated edges, and the storage overhead of the target data entity as input parameters to calculate a comprehensive score; according to the predefined numerical range of the comprehensive score, the target data entity is allocated to the hot layer, warm layer, or cold layer;

[0029] S3.3: When any data entity is migrated to the hot layer according to the decision, based on the relational adjacency index, retrieve the associated data entities that have relational connection edges with the data entities in the hot layer and are currently located in the warm or cold layer, and preload the associated data entities into the preparation buffer of the warm layer.

[0030] S3.4: After the data entity migrates between cache layers according to S3.2-S3.3, the storage location status of the corresponding data entity in the heterogeneous cache graph is updated synchronously, and the entries associated with the corresponding data entity in the multi-dimensional index are updated.

[0031] Specifically, S3 also includes an access-driven graph heat update mechanism, which specifically includes:

[0032] S3.5: When a data entity in the cache is successfully accessed, update the real-time heat entropy of the corresponding data entity based on the timestamp of the access event;

[0033] S3.6: Trigger related popularity propagation based on the operation type of the access request, including:

[0034] If the operation type is to read the attribute of the data entity, then the real-time heat entropy of the corresponding data entity is propagated to the attribute node directly connected to it.

[0035] If the operation type is to traverse the relationship edges between the data entity and non-self data entities, then the real-time heat entropy of the corresponding data entity is propagated in a decaying manner along the traversed relationship edges to the associated data entities.

[0036] S3.7: Update the heat changes generated in steps S3.5 and S3.6 to the time-series heat index in real time;

[0037] S3.8: Re-evaluate the cache location of data entities based on the updated time-series popularity index;

[0038] If the real-time heat entropy change of any data entity causes the comprehensive score of its cache location decision function to cross the predefined hierarchical threshold, then the corresponding data entity is triggered to migrate between the hot layer, the warm layer, and the cold layer.

[0039] Specifically, based on the association and confidence characteristics of each entity in the heterogeneous cache graph, labeled data is identified, data correction suggestions are generated and fed back to the incremental cleaning process to update the structured labeled data stream and the heterogeneous cache graph, including:

[0040] S4.1: In the heterogeneous cache graph, maintain a confidence trajectory for each attribute node; the confidence trajectory is used to record the attribute node identifier, historical values, value source, frequency of occurrence, last update timestamp, and value confidence.

[0041] S4.2: Based on the heterogeneous cache graph, a predefined entity comprehensive confidence evaluation model is used to evaluate each data entity; the entity comprehensive confidence evaluation model aggregates the consistency of the confidence trajectory of all attribute nodes of the target data entity, the diversity of the sources of its association connection edges, and the stability characteristics of the corresponding data entity in historical correction, and outputs an entity confidence coefficient between 0 and 1.

[0042] S4.3: Periodically scan the heterogeneous cache map. When the entity confidence coefficient of any data entity is lower than the preset confidence alarm threshold, mark the corresponding data entity as a low-confidence entity and automatically generate a source tracing and correction proposal. The source tracing and correction proposal includes at least the conflicting attribute values, the comparison of values ​​from different sources, and the recommended correction values ​​based on statistical and correlation analysis.

[0043] S4.4: The generated source tracing correction proposal is encapsulated as a correction event and fed back to the incremental cleaning process. The incremental cleaning process updates the structured labeled data stream based on the correction event, triggering the update of the heterogeneous cache map.

[0044] Specifically, in step S4, corresponding subgraph copies of the heterogeneous cached graph are maintained locally on each data source node, and the graph neural network model parameters and hotspot entity states of each subgraph copy are synchronized through a secure aggregation algorithm, including:

[0045] S4.5: Each participating data source node, based on its data jurisdiction, cuts out a corresponding subgraph copy from the global dynamic entity association graph for maintenance; the subgraph copy is a slice of the heterogeneous cache graph;

[0046] S4.6: Establish a synchronization cycle. Within each cycle, each data source node executes:

[0047] The time-series graph neural network is trained using local data, and the gradients of the model parameters calculated after training are homomorphically encrypted to form encrypted gradient data.

[0048] Extract the top K data entities with the highest real-time heat entropy from the subgraph replica, obtain the entity confidence coefficient and attributes of the data entities, form a local hot entity state snapshot, and encrypt the local hot entity state snapshot;

[0049] The encrypted gradient data and the encrypted snapshot of the local hotspot entity status are uploaded to the central coordinator.

[0050] Specifically, in step S4, maintaining corresponding subgraph copies of the heterogeneous cached graph locally on each data source node, and synchronizing the graph neural network model parameters and hotspot entity states of each subgraph copy using a secure aggregation algorithm, further includes:

[0051] S4.7: After receiving the encrypted information uploaded by all data source nodes, the central coordinator executes a secure aggregation algorithm, including: performing a secure average calculation on the encrypted gradient data from all data source nodes, decrypting the calculation result, and using the decrypted calculation result to update the graph neural network model, which is a temporal graph neural network; simultaneously, merging the encrypted local hotspot entity state snapshots from all data source nodes, analyzing to obtain the global hotspot and confidence distribution status; and distributing the updated graph neural network model parameters to all data source nodes.

[0052] S4.8: Each data source node receives the graph neural network model parameters issued by the central coordinator, updates its local time-series graph neural network, and adjusts the state of relevant data entities in the subgraph replicas.

[0053] Specifically, the specific steps of S5 include:

[0054] S5.1: When an access request for a target data entity is received, the subject identity, operation type, and environment token are extracted from the context of the access request;

[0055] S5.2: Query the heterogeneous cache graph to obtain all attribute nodes of the target data entity, associated data entities and their relationship types, and dynamically generate the context feature vector of this access by combining the historical access records of the subject identity;

[0056] S5.3: Input the generated context feature vector into the deep policy evaluation network and output an arbitration decision; the arbitration decision is one of allow, deny, or downgrade, wherein the downgrade decision means that only the de-identified simplified version of the target data entity stored in the cold layer is allowed to access;

[0057] S5.4: Enforcing the arbitration decision, including:

[0058] If the arbitration decision permits, then access rights shall be granted;

[0059] If the arbitration decision is to downgrade or reject, the access event is first recorded, and then a cache downgrade instruction for the target data entity is sent to the hierarchical caching architecture. The cache downgrade instruction is used to force the value of the target data entity to zero when updating the real-time heat entropy.

[0060] Compared with the prior art, the beneficial effects of the present invention are:

[0061] 1. This invention proposes an intelligent caching method for real-time labeled data streams. This method achieves accurate mapping and incremental cleaning of multi-source labeled fields through a temporal graph neural network. Combined with a heterogeneous cache graph and a four-element composite index, it efficiently integrates data entities, attributes, and relationships, improving data structuring efficiency and retrieval flexibility. The hierarchical caching architecture dynamically allocates data based on real-time heat entropy and relationship strength, coupled with preloading of associated entities and an access-driven heat propagation mechanism, optimizing cache resource utilization and data access response speed. At the same time, through entity confidence assessment and source tracing correction feedback, it continuously iterates and optimizes data quality, ensuring the accuracy and consistency of labeled data.

[0062] 2. This invention proposes an intelligent caching method for real-time labeled data streams. This method leverages a secure aggregation algorithm to achieve encrypted collaboration of distributed data sources, maintaining subgraph copies locally on each node and synchronizing model parameters and hot entity states, thus balancing the efficiency of distributed processing with the security of data transmission. Furthermore, fine-grained dynamic access control based on a deep policy evaluation network can make precise decisions based on subject identity, operation type, and data association characteristics, supporting access permission, denial, or de-identification and degradation. This ensures both the compliance and security of data access and optimizes the caching layering strategy through control decision feedback, further enhancing the collaboration and reliability of the overall processing flow. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the intelligent caching processing method for real-time labeled data streams according to the present invention;

[0064] Figure 2 This is a flowchart illustrating the principle of the intelligent caching processing method for real-time labeled data streams according to the present invention. Detailed Implementation

[0065] The technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the application scenarios in the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0066] The intelligent caching processing method for real-time labeled data streams provided in this application is applicable to real-time processing scenarios of multi-source heterogeneous labeled data, and is especially suitable for fields such as intelligent transportation, industrial IoT, and smart security where data real-time performance, accuracy, and security requirements are high. This embodiment uses multi-source labeled data stream processing of intelligent transportation systems as the main thread, and elaborates on the specific implementation process of the method. In this intelligent transportation scenario, the data sources involved include five core data sources: roadside high-definition cameras, millimeter-wave radar, traffic signal controllers, vehicle terminals, and electronic police systems. The real-time labeled data includes vehicle entities, pedestrian entities, road entities, and traffic event entities. Among them, vehicle entities include license plates, speed, vehicle type, and driving trajectory; pedestrian entities include identity characteristics, walking direction, and crossing status; road entities include road segment numbers, number of lanes, and road surface conditions; and traffic event entities include congestion level, accident type, and violations. The real-time data inflow rate is 500-800 records / second, requiring a cache hit rate of no less than 95%, a data processing latency of no more than 20ms, and a data accuracy of no less than 98%. At the same time, it is necessary to ensure the access security of different authorized entities, such as traffic management departments, third-party data analysts, and the general public.

[0067] Please see Figure 1 and Figure 2 This embodiment provides an intelligent caching method for real-time labeled data streams, including the following steps:

[0068] S1: Receives multi-source labeled data streams, maps labeled fields from different sources to nodes of a temporal graph neural network, calculates semantic and structural similarity between nodes, generates field mapping rules and performs incremental cleaning, and outputs structured labeled data streams.

[0069] S2: Using the structured labeled data stream as input, extract the data entities, entity attributes, and relationships between entities contained therein, construct a heterogeneous cache graph, and establish a multi-dimensional index;

[0070] S3: Based on the heterogeneous cache graph, construct a hierarchical cache architecture that includes a hot layer, a warm layer and a cold layer, dynamically allocate data entities in the heterogeneous cache graph to the corresponding cache layer, and synchronously update the heterogeneous cache graph status and multi-dimensional index in real time.

[0071] S4: Based on the association and confidence characteristics of each data entity in the heterogeneous cache graph, identify the labeled data, generate data correction suggestions and feed them back to the incremental cleaning process to update the structured labeled data stream and the heterogeneous cache graph. At the same time, maintain the corresponding subgraph copy of the heterogeneous cache graph locally on each data source node, and synchronize the graph neural network model parameters and hot spot entity status of each subgraph copy through a secure aggregation algorithm.

[0072] S5: Based on the attributes and association features of data entities in the heterogeneous cache graph, perform fine-grained dynamic access control on the data stored in the hierarchical cache architecture, and feed the control decisions back to S3.

[0073] Regarding S1 above:

[0074] The core objective of S1 is to address issues such as heterogeneous data fields, inconsistent formats, and noise in multi-source labeled data. It achieves data standardization and incremental cleaning through node mapping and rule generation using a time-series graph neural network. In this embodiment, the labeled fields of various data sources differ significantly: for example, the vehicle speed field output by roadside HD cameras is named veh_speed, in km / h, and in floating-point format; the target speed field output by millimeter-wave radar is named tgt_vel, in m / s, and in integer format; the driving speed field output by vehicle-mounted terminals is named driving_speed, in km / h, and in floating-point format; the signal cycle field output by traffic light controllers is named signal_cycle, in seconds, and in integer format; and the speed field of violating vehicles output by electronic police systems is named viol_veh_spd, in km / h, and in floating-point format. Furthermore, some data sources contain missing data or outliers, such as negative vehicle speeds, requiring incremental cleaning to purify the data.

[0075] S1 specifically includes:

[0076] S1.1: Create a temporal feature node for each labeled field from each source; the attributes of the temporal feature node include the semantic embedding vector of the field name, the statistical feature vector of the value range distribution, and the timestamp sequence of field occurrence and change events;

[0077] In this embodiment, the semantic embedding vector of field names is encoded using a BERT-base-uncased pre-trained model, outputting a 768-dimensional semantic embedding vector. For example, encoding veh_speed yields a vector [0.021, 0.035, ..., 0.018] with a total of 768 dimensions, and encoding tgt_vel yields a vector [0.019, 0.042, ..., 0.023]. The semantic embedding vector can quantify the semantic similarity of field names.

[0078] For the statistical feature vector of the value range distribution, statistical analysis is performed on the historical data of each field, such as the labeled data of the last hour, to extract the mean, variance, minimum, maximum, median, quartiles, and missing data rate to form the statistical feature vector. For example, the historical data statistical results of the veh_speed field of the roadside high-definition camera are: mean 55.3, variance 89.2, minimum 0.0, maximum 120.0, median 53.5, first quartile 38.2, third quartile 68.7, and missing data rate 0.3%. The corresponding statistical feature vector is [55.3, 89.2, 0.0, 120.0, 53.5, 38.2, 68.7, 0.003]. The calculation formulas of mean, variance, minimum, maximum, median, quartiles, and missing data rate are all existing technologies in this field and are not the inventive solutions of this application, and will not be elaborated here.

[0079] For the timestamp sequence of field occurrence and change events, by recording the timestamps of the most recent 100 occurrences and the most recent 10 changes of the field, such as field format adjustments or changes in collection frequency, for example, the timestamp sequence of the veh_speed field is [1698765432100, 1698765432150, ..., 1698765437050] and [1698760000000,1698762000000, ..., 1698765000000], we can obtain 100 occurrence timestamps and 10 change timestamps.

[0080] In this embodiment, the time-series feature nodes are stored in key-value pairs. The key is the data source identifier-field name, such as camera_01-veh_speed, and the value is a JSON object containing the above three types of attributes, which is temporarily stored in the Redis cache for easy reading and updating.

[0081] S1.2: Input all temporal feature nodes within the continuous time window into the temporal graph neural network. The temporal graph neural network captures the temporal dependency features of each temporal feature node through the time-gated recurrent unit layer, and then aggregates the cross-source structural features between temporal feature nodes from different data sources through the graph attention layer, and outputs the fused feature vector of each temporal feature node.

[0082] In this embodiment, the temporal graph neural network is implemented using the PyTorch framework. The network structure includes an input layer, a temporally gated recurrent unit (GRU) layer, a graph attention layer, and an output layer. The specific parameter configurations are as follows: the input layer has a dimension of 776, containing a 768-dimensional semantic embedding vector and an 8-dimensional statistical feature vector; the temporally gated recurrent unit layer contains two hidden layers, each with 256 neurons, using ReLU activation function and a dropout rate of 0.2, to capture the temporal dependency features of each temporal feature node; the graph attention layer contains one hidden layer with 128 neurons and four attention heads, using a concatenation method to fuse the outputs of multiple attention heads, to aggregate cross-source structural features between nodes from different data sources; the output layer has a dimension of 128, outputting the fused feature vector of each temporal feature node.

[0083] In the specific implementation, the continuous time window length is set to 10 seconds, that is, the window slides once every 10 seconds, and all temporal feature nodes within the window are input into the temporal graph neural network. In this embodiment, there are 32 labeled fields from 5 types of data sources, corresponding to 32 temporal feature nodes. First, the time-gated recurrent unit layer processes the timestamp sequence of each temporal feature node: taking the camera_01-veh_speed node as an example, the time-gated recurrent unit layer focuses on the time dependency relationship of the last 5 occurrences of data through the time gating mechanism, converting the 776-dimensional input vector into a 256-dimensional time dependency feature vector; then, the graph attention layer calculates the attention weights of this node with the remaining 31 nodes. For example, the attention weight with the radar_01-tgt_vel node is 0.85, and the attention weight with the signal_01-signal_cycle node is 0.03. The time dependency feature vectors of all nodes are weighted and aggregated according to the attention weights, and finally a 128-dimensional fused feature vector is output.

[0084] In this embodiment, the training process of the temporal graph neural network adopts an offline pre-training and online fine-tuning approach: in the offline stage, multi-source labeled data from the past month is used for pre-training, the optimizer is AdamW, the learning rate is set to 1e-4, the training epochs are 50, and the loss function is mean squared error loss; in the online stage, the latest 10 minutes of data are used for fine-tuning every hour, and the learning rate is adjusted to 5e-5 to ensure that the network can adapt to the dynamic changes in data distribution.

[0085] S1.3: Calculate the cross-source semantic affinity between the fused feature vectors of any two temporal feature nodes, and calculate the temporal co-occurrence intensity based on the co-occurrence relationship of the two temporal feature nodes within the sliding time window. Then, input the cross-source semantic affinity and the temporal co-occurrence intensity into the rule generation neural network. The rule generation neural network outputs a probabilistic field mapping rule matrix, where each element in the field mapping rule matrix represents the confidence weight of mapping one source field to another source field.

[0086] Furthermore, the temporal co-occurrence intensity is calculated based on the co-occurrence relationship of two temporal feature nodes within a sliding time window, including:

[0087] (1) Based on the current time, a sliding time window is defined backward according to a preset length. The timestamps of all events occurring within the sliding time window of the first time-series feature node and the second time-series feature node are extracted respectively to form the first node timestamp sequence and the second node timestamp sequence.

[0088] (2) For the defined sliding time window, count the number of co-occurrence event pairs between the first node timestamp sequence and the second node timestamp sequence, wherein a co-occurrence event pair is defined as any two timestamp combinations that come from the first node timestamp sequence and the second node timestamp sequence respectively and whose absolute time difference is less than a preset time difference threshold.

[0089] (3) Calculate the ratio of the number of co-occurring event pairs to the geometric mean of the total number of events in the first node timestamp sequence and the total number of events in the second node timestamp sequence, and use this ratio as the local co-occurrence confidence of the current sliding time window;

[0090] (4) Assign a time decay weight to each sliding time window in the past and present. The closer the window is to the current time, the greater its time decay weight. Then multiply the local co-occurrence confidence of all sliding time windows by their respective time decay weights, and sum all the product results to obtain the weighted aggregate value.

[0091] (5) Divide the obtained weighted aggregate value by the sum of the time decay weights of all sliding time windows, normalize it, and finally output a value between 0 and 1 as the temporal co-occurrence intensity.

[0092] In this embodiment, cross-source semantic affinity is calculated using cosine similarity, which is the dot product of the fused feature vectors of two temporal feature nodes divided by the product of the magnitudes of the two fused feature vectors. The value ranges from [-1, 1], with values ​​closer to 1 indicating greater semantic similarity. For example, the cosine similarity of the fused feature vectors of camera_01-veh_speed and radar_01-tgt_vel is 0.92, the cosine similarity with obd_01-driving_speed is 0.88, the cosine similarity with e_police_01-viol_veh_spd is 0.85, and the cosine similarity with signal_01-signal_cycle is 0.05.

[0093] The temporal co-occurrence intensity is calculated based on the number of times two nodes co-occur within a sliding time window and the time interval between them: the more times they co-occur and the shorter the time interval, the higher the temporal co-occurrence intensity, with a value range of [0,1]. For example, camera_01-veh_speed and radar_01-tgt_vel co-occur 8 times within a 10-second window, with an average time interval of 0.5 seconds, resulting in a temporal co-occurrence intensity of 0.90; it co-occurs 6 times with obd_01-driving_speed, with an average time interval of 1.2 seconds, resulting in a temporal co-occurrence intensity of 0.75; and it co-occurs 2 times with signal_01-signal_cycle, with an average time interval of 4.8 seconds, resulting in a temporal co-occurrence intensity of 0.12.

[0094] The rule-generating neural network is also implemented using PyTorch. The input layer has a dimension of 2, representing both cross-source semantic affinity and temporal co-occurrence strength. The hidden layer contains two fully connected layers with 64 and 32 neurons respectively, using ReLU activation. The output layer has a dimension of 1, and the output value is a confidence weight in the interval [0,1], representing the reliability of mapping one source field to another. The calculated cross-source semantic affinity and temporal co-occurrence strength are input into the rule-generating neural network, and the output is a field mapping rule matrix. In this embodiment, the field mapping rule matrix is ​​a 32×32 matrix because there are 32 labeled fields. An example of the matrix elements is shown below:

[0095] The confidence weight of camera_01-veh_speed-radar_01-tgt_vel is 0.91;

[0096] The confidence weight of camera_01-veh_speed-obd_01-driving_speed is 0.87;

[0097] The confidence weight of camera_01-veh_speed-e_police_01-viol_veh_spd is 0.83;

[0098] The confidence weights of the non-speed fields camera_01-veh_speed are all below 0.1.

[0099] S1.4: Based on the field mapping rule matrix, the original labeled fields that flow in in real time are parsed and aligned. Data with a mapping confidence level higher than a preset first threshold are directly fused. Data with a confidence level lower than the preset first threshold but higher than the preset second threshold are marked and temporarily stored in the pending area. Data with a confidence level lower than the preset second threshold is triggered with an anomaly alarm. Finally, a structured labeled data stream with uniform fields and standardized format is output. In this embodiment, the preset first threshold is 0.8 and the second threshold is 0.3.

[0100] Optionally, to address the issue of high noise in some data sources, such as older sensors, in scenarios like the Industrial Internet of Things (IIoT), a noise filtering layer can be added in S1.2. Specifically, before inputting into the GRU layer, outlier detection and filtering are performed on the statistical feature vectors of time-series feature nodes. An isolated forest algorithm is used to identify outliers in the statistical features, such as abnormal value ranges or sudden changes in missing rates, and these outliers are replaced with the median of the field. Simultaneously, a noise weight factor is introduced into the graph attention layer, setting a lower initial attention weight (e.g., 0.5) for data source nodes with high noise to reduce their impact on the fused feature vectors. In this embodiment, if the millimeter-wave radar's data noise variance is three times higher than normal equipment due to hardware aging, this optimization scheme improves the average confidence weight of its field mapping with other data sources by 0.08, and the accuracy after data fusion by 3.2%. The isolated forest algorithm is existing technology in this field and is not an inventive solution of this application; therefore, it will not be elaborated upon here.

[0101] Optionally, for scenarios with significant differences in data update frequencies, such as a camera data update frequency of 20Hz and a traffic light data update frequency of 1Hz, the time window length can be dynamically adjusted. Specifically, this is achieved by calculating the average update cycle of each data source, using 10 times the greatest common divisor of all data source update cycles as the base window length, and then dynamically adjusting it based on the data volatility, i.e., variance: when data volatility is high (i.e., variance is greater than or equal to a preset variance threshold), the window length is shortened by 30% to improve the response speed to data changes; when data volatility is low (i.e., variance is less than the preset variance threshold), the window length is extended by 50% to reduce computational overhead. In this embodiment, the camera data update cycle is 50ms, radar is 100ms, and traffic light is 1000ms, the greatest common divisor is 50ms, and the base window length is 500ms; during peak hours when vehicle data volatility is high, the window length is adjusted to 350ms, and during off-peak hours it is adjusted to 750ms. Compared to a fixed window length, this improves computational efficiency and reduces data latency.

[0102] Regarding S2 above:

[0103] In this embodiment, the heterogeneous cache graph takes the core entities in the intelligent transportation scenario as the center and constructs a multi-relationship graph structure that includes entities, attributes, and relationships. The multi-dimensional index is designed to meet the query needs of the graph and enable fast retrieval and positioning.

[0104] The process of constructing the heterogeneous cache map in S2 includes:

[0105] S2.1: Identify atomic-level data entities from the structured labeled data stream and assign a globally unique entity fingerprint to each data entity; the entity fingerprint is generated by combining its entity type hash value and attribute hash value;

[0106] Furthermore, the identification of atomic-level data entities adopts a combination of rule-based and machine learning approaches: First, basic entity types are identified through preset rules, including vehicles, pedestrians, roads, and traffic events. For example, identifiers starting with "veh_" represent vehicle entities, those starting with "ped_" represent pedestrian entities, those starting with "road_" represent road entities, and those starting with "event_" represent traffic event entities. Then, the core attributes of entities are extracted from the structured labeled data stream using the BERT model to determine atomic-level data entities, i.e., the smallest indivisible entity units. The BERT model is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0107] Furthermore, the entity fingerprint is generated using the SHA-256 hash algorithm. Specifically, the entity type is first hashed to obtain a 256-bit entity type hash value; then, the core attribute key-value pairs of the entity are sorted lexicographically and concatenated, and hashed to obtain a 256-bit attribute hash value; finally, the entity type hash value and the attribute hash value are concatenated to form a 512-bit entity fingerprint, serving as a globally unique identifier for the entity. For example, the entity fingerprint of the vehicle entity veh_10086 is “7a3f9d...2b8e”, ensuring that even if different data sources identify the same entity differently, a unique match can be achieved through the entity fingerprint. The SHA-256 hash algorithm is existing technology in this field and is not an inventive solution of this application; therefore, it will not be elaborated upon here.

[0108] S2.2: Extract the attribute key-value pairs of each data entity, instantiate each attribute key-value pair into an attribute node, and link each attribute node to its data entity through an edge;

[0109] In this embodiment, each data entity's attribute key-value pair is instantiated as an independent attribute node. The identifier of the attribute node is the entity fingerprint code-attribute name, such as "7a3f9d...2b8e-veh_speed". The attributes of the attribute node include the attribute value, data source, update timestamp, and confidence level. For example, the veh_speed attribute node of the vehicle entity veh_10086 has an attribute value of 65.3 km / h, a data source of "camera_01+radar_01+obd_01", an update timestamp of 1698765438200, and a confidence level of 0.93.

[0110] Furthermore, attribute nodes are linked to their respective data entities via "belonging" edges, the attributes of which include the link establishment timestamp and the attribute update frequency. For example, the "belonging" edge between the veh_10086 entity and the veh_speed attribute node has an establishment timestamp of 1698765432000 and an attribute update frequency of 20Hz.

[0111] S2.3: Identify the relationships between different data entities, instantiate each relationship as a directed relationship edge, and assign the relationship edge a relationship type, establishment time, and a relationship strength value based on co-occurrence frequency and semantic consistency;

[0112] Furthermore, the relationships between different data entities are identified through rule-based and statistical methods: common relationship rules are preset, such as vehicle-passing-road, pedestrian-crossing-lane, traffic incident-occurring-road, vehicle-violation-traffic incident, and the co-occurrence frequency of two entities in time and space is counted. If the co-occurrence frequency is higher than the preset frequency threshold and meets the rules, it is determined that there is a relationship. In this embodiment, the preset frequency threshold is set to 5 times / minute.

[0113] Furthermore, the relational connection edge is instantiated as a directed edge, and the edge is identified as source entity fingerprint code-relationship type-target entity fingerprint code, such as "7a3f9d...2b8e-pass-RD001...5c7d". The edge attributes include relation type, establishment time, and relation strength value. The relation strength value is calculated by weighting co-occurrence frequency and semantic consistency. The co-occurrence frequency weight is 0.6, the semantic consistency weight is 0.4, and the value range is [0,1]. For example, the "passing" relationship between vehicle entity veh_10086 and road entity road_01 has a co-occurrence frequency of 12 times / minute and a semantic consistency of 0.95. Therefore, the relationship strength value is 0.6×(12 / 20)+0.4×0.95=0.36+0.38=0.74, where the co-occurrence frequency is normalized to the [0,1] interval, and the maximum co-occurrence frequency is 20 times / minute. The "crossing" relationship between pedestrian entity ped_001 and lane entity lane_03 has a co-occurrence frequency of 3 times / minute and a semantic consistency of 0.90. Therefore, the relationship strength value is 0.6×(3 / 20)+0.4×0.90=0.09+0.36=0.45.

[0114] S2.4: Based on all identified data entities, all generated attribute nodes and their links to data entities, and all generated relational links, construct a heterogeneous cache graph. The heterogeneous cache graph uses data entities as vertices, attribute nodes as subordinate vertices, and links between data entities and attribute nodes, as well as relational links between data entities, as edges, forming a multi-relationship graph structure with attributes.

[0115] Furthermore, the heterogeneous cache graph constructed in this embodiment is a multi-relationship graph structure with attributes, specifically configured as follows:

[0116] Vertices: These include data entity vertices and attribute nodes. The attributes of data entity vertices include entity fingerprint, entity type, core attribute summary, and creation timestamp; the attributes of attribute nodes include attribute value, data source, update timestamp, and confidence level.

[0117] Edges include "belonging" edges and relational edges. The attributes of "belonging" edges include establishment time and update frequency; the attributes of relational edges include relation type, establishment time, and relation strength value. "Belonging" edges refer to the edges between data entities and attribute nodes, while relational edges refer to the edges between data entities.

[0118] Furthermore, the heterogeneous cache graph is stored using the Neo4j graph database. Neo4j supports efficient graph queries and relation traversal, meeting the needs of real-time processing scenarios. To improve storage efficiency, duplicate attributes in the heterogeneous cache graph, such as data source names and relation type names, are encoded for storage. For example, camera_01 is encoded as C01, and the pass relation is encoded as R01. Decoding is then performed during queries.

[0119] The multi-dimensional index in S2 is a four-element composite index structure, which includes an attribute inverted index, a relation adjacency index, a time-series heat index, and a subgraph structure fingerprint index.

[0120] In this embodiment, the four-element composite index structure is built on Elasticsearch, and utilizes Elasticsearch's distributed search capabilities to achieve efficient multi-dimensional retrieval. Elasticsearch is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0121] The attribute inverted index is an inverted list built for the keys and values ​​of all attribute nodes. It is used for fuzzy matching and range queries based on attribute values. For example, the inverted list corresponding to the attribute key veh_speed contains the veh_speed attribute node identifier and its attribute value for all vehicle entities. It can quickly query all vehicle entities with "veh_speed > 60km / h" or perform fuzzy queries on related vehicle entities. In this embodiment, the fields of the attribute inverted index include attribute key, attribute value, attribute node identifier, and entity fingerprint. The index has 5 shards and 2 replicas to ensure high availability and performance of queries.

[0122] The relational adjacency index records the relation type, target data entity, and current relation strength value of all relational edges connecting each data entity, and is sorted in descending order of relation strength value. For example, the relational adjacency index records for the vehicle entity veh_10086 are: "R01 (pass) - RD001...5c7d (road_01) - 0.74", "R02 (violate) - EV005...8a9b (event_05) - 0.62", and "R03 (near) - veh_10092 (veh_10092) - 0.58", ordered in descending order of relation strength value. The fields of the relational adjacency index include the source entity fingerprint, relation type code, target entity fingerprint, and relation strength value. It uses Elasticsearch's nested type storage and supports quick filtering of target data entities by source entity fingerprint and relation type.

[0123] The time-series heat index is based on the event timestamps of when a data entity is accessed, associated, or its attributes are modified. It uses a time decay function to dynamically calculate the real-time heat entropy of each data entity and establishes a sorted index of the real-time heat entropy.

[0124] In this embodiment, the time decay function adopts an exponential decay model, and the specific formula is: the real-time heat entropy is equal to the product of the base heat value and the first variable, where the first variable is exp(-λ×Δt), where λ is the decay coefficient, which is 0.001 in this embodiment, and Δt is the time difference between the current time and the event timestamp; the base heat value is set according to the event type: the base heat value of the data entity being accessed is 10, the base heat value of the attribute being modified is 8, and the base heat value of the associated entity is 5. For example, vehicle entity veh_10086 was accessed 10 seconds ago (Δt=10), with a base popularity value of 10 and a popularity entropy of 10×exp(-0.001×10)=9.90; its attribute was modified 5 seconds ago (Δt=5), with a base popularity value of 8, resulting in a cumulative popularity entropy of 9.90+8×exp(-0.001×5)=9.90+7.96=17.86; and it was associated with another entity 3 seconds ago (Δt=3), with a base popularity value of 5, resulting in a final popularity entropy of 17.86+5×exp(-0.001×3)=17.86+4.98=22.84. The time-series popularity index includes fields such as entity fingerprint, real-time popularity entropy, and recent event timestamp. It uses Elasticsearch's sorted index, supports querying hot entities in descending order of real-time popularity entropy, and has a query response time of no more than 5ms.

[0125] The subgraph structure fingerprint index calculates the local subgraph structure within two hops centered on each data entity, generating a fixed-length subgraph structure fingerprint for graph-based similarity retrieval.

[0126] In this embodiment, the subgraph structure fingerprint generation process includes: extracting all nodes and edges within two hops centered on the target data entity; encoding the node type and edge type, such as encoding a vehicle entity as 1, a road entity as 2, a "pass" relationship as 1, and a "belong" relationship as 2; and using a graph hashing algorithm to perform hash calculation on the encoded subgraph structure to generate a 256-bit fixed-length fingerprint. For example, two vehicle entities traveling on the same road with similar speeds and vehicle types have similar subgraph structures within two hops, and the Hamming distance of their subgraph structure fingerprints is less than 32. The graph hashing algorithm is prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0127] Furthermore, the subgraph structure fingerprint index includes fields such as entity fingerprint code and subgraph structure fingerprint, which are stored using Elasticsearch's keyword type. It supports fingerprint-based similarity queries and can quickly retrieve data entities with similar relational structures.

[0128] Optionally, to address the issue of query performance degradation caused by the increasing size of the graph over time, dynamic pruning can be performed on the heterogeneous cached graph. Specifically, this involves setting pruning thresholds, including a relationship strength threshold and an attribute node update time threshold. In this embodiment, the relationship strength threshold is 0.2, and the attribute node update time threshold is 7 days. The graph is scanned daily at 2 AM, deleting relationship edges with a relationship strength value lower than 0.2 and attribute nodes that have not been updated for more than 7 days and have a confidence level lower than 0.5. Simultaneously, the multi-dimensional indexes involved in the deleted edges and nodes are updated synchronously. In this embodiment, through dynamic pruning, graph storage is reduced by 35%, relationship traversal query speed is improved by 40%, and the availability of core data is not affected.

[0129] Optionally, to address the issue of the long update time for a full index, an incremental update mechanism is adopted. Specifically, when updates to the structured labeled data stream cause changes to the heterogeneous cached graph (e.g., adding entities, updating attribute values, or adding relationships), only the indexes corresponding to the changed parts are updated, without rebuilding the entire index. For example, if the veh_speed attribute value of the vehicle entity veh_10086 is updated to 68.5 km / h, only the attribute value field of that attribute node in the attribute inverted index, the heat entropy of that entity in the time-series heat index, and the most recent event timestamp are updated; unchanged index fields remain unchanged. In this embodiment, the incremental update takes only 1 / 20th the time of a full update, reducing the overhead of index maintenance and ensuring real-time synchronization between the index and the graph.

[0130] Regarding S3 above:

[0131] In this embodiment, the storage media of the hierarchical caching architecture are selected differently according to performance requirements, and the cache allocation and migration strategy is dynamically adjusted based on factors such as the real-time popularity of entities, relationship strength, and storage overhead.

[0132] The specific steps of S3 include:

[0133] S3.1: Divide the cache storage medium into three logical layers, including a hot layer, a warm layer, and a cold layer;

[0134] Furthermore, the hardware configuration, performance indicators, and applicable scenarios for each layer are as follows:

[0135] Hot Layer: Employs an all-flash array, with each cache node configured with a 2TB SSD, 16 cores, and 32GB of memory. Deployed in a local data center close to the data processing nodes, it achieves access latency of less than 1ms and throughput greater than 1000MB / s. Suitable for storing hot entities with high real-time heat entropy, high access frequency, and high relationship strength, such as vehicles currently traveling on main roads or ongoing traffic events.

[0136] Temperature Layer: Employs a hybrid storage array, with each cache node configured with a 10TB SATA HDD, 8 cores, 16GB of memory, and 200GB of SSD cache. Deployed in a regional data center, it achieves an access latency of 1-10ms and a throughput greater than 500MB / s. Suitable for storing entities with moderate real-time heat entropy, moderate access frequency, and a certain degree of correlation, such as vehicles that have traveled within the past hour or traffic events that have occurred within the past two hours.

[0137] Cold Layer: Employs object storage, with cloud storage capacity expandable on demand. The local tape library is configured with 100TB of storage capacity and deployed in a remote data center. Access latency is greater than 10ms, and throughput is greater than 100MB / s. Suitable for storing entities with low real-time heat entropy, low access frequency, and only requiring long-term archiving, such as vehicle entities from 24 hours ago or completed traffic event entities.

[0138] Each cache layer is uniformly scheduled through Kubernetes, and a distributed cache management system is used to achieve load balancing and failover of cache nodes.

[0139] S3.2: Based on the heterogeneous cache map, a cache location decision function is used to dynamically allocate data entities in the heterogeneous cache map to the divided cache layers. The cache location decision function takes the real-time heat entropy of the target data entity, the average relationship strength value of its associated edges, and the storage cost of the target data entity as input parameters, and calculates a comprehensive score by weighted summation. According to the predefined numerical range of the comprehensive score, the target data entity is allocated to the hot layer, warm layer, or cold layer.

[0140] In this embodiment, the predefined numerical range is as follows:

[0141] Thermal layer: Overall score > 0.8;

[0142] Thermosphere: 0.3 ≤ Overall score ≤ 0.8;

[0143] Cold layer: Overall score <0.3;

[0144] In this embodiment, the comprehensive score of the vehicle entity veh_10086 is 0.743, which falls within the 0.3-0.8 range, and therefore it is assigned to the thermal layer. If the real-time thermal entropy of this entity increases to 50, its normalized value = 50 / (50+10) = 0.833, where 10 in the denominator is the preset thermal entropy baseline value. The average relationship strength value of the associated connection edges remains at 0.65, and the storage overhead remains at 5000 bytes. The comprehensive score = 0.833×0.5+0.65×0.3+0.9951×0.2 = 0.4165+0.195+0.199≈0.8105, which exceeds 0.8, and it is assigned to the thermal layer. Here, 0.9951 is the result after normalization of 5000 bytes, based on the actual storage overhead and the preset minimum storage space for the scenario. The maximum storage overhead of the entity is calculated based on the sales and preset scenarios. If the real-time heat entropy of the entity drops to 5, its normalized value = 5 / (5+10) = 0.333, the average relationship strength value of the associated edge drops to 0.25, and the comprehensive score = 0.333×0.5+0.25×0.3+0.9951×0.2 = 0.1665+0.075+0.199≈0.4405, still in the warm layer. If the real-time heat entropy drops to 1, its normalized value = 1 / (1+10) = 0.091, the average relationship strength value of the associated edge drops to 0.15, and the comprehensive score = 0.091×0.5+0.15×0.3+0.9951×0.2 = 0.0455+0.045+0.199≈0.2895, which is lower than 0.3, and it is assigned to the cold layer.

[0145] S3.3: When any data entity is migrated to the hot layer according to the decision, based on the relational adjacency index, retrieve the associated data entities that have a high-strength relational connection edge with the hot layer data entity and are currently located in the warm or cold layer, and preload the associated data entities into the preparatory buffer of the warm layer.

[0146] For example, vehicle entity veh_10086 is moved to the hot layer because its comprehensive score reaches 0.8105. By querying the relational adjacency index, we can find related data entities with a relational strength value greater than 0.6: road entity "road_01" with a relational strength value of 0.74, currently located in the warm layer, and traffic event entity "event_05" with a relational strength value of 0.62, currently located in the warm layer. These two related data entities are preloaded into the warm layer's preparation buffer. When there is a request to access these two data entities, they can be read directly from the preparation buffer, reducing access latency by 60%.

[0147] Furthermore, if the associated data entity is currently located in the cold layer, such as a road maintenance record entity that has not been accessed for a long time but is closely related to the data entity in the hot layer, it will be migrated from the cold layer to the warm layer preparation buffer, and its storage location status and multi-dimensional index in the heterogeneous cache map will be updated synchronously.

[0148] S3.4: After the data entity migrates between cache layers according to S3.2-S3.3, the storage location status of the corresponding data entity in the heterogeneous cache graph is updated synchronously, and the entries associated with the corresponding data entity in the multi-dimensional index are updated.

[0149] Furthermore, the heterogeneous cache graph state update includes: in Neo4j, adding a "storage_layer" attribute to each data entity vertex, with values ​​of "hot", "warm", and "cold". When an entity migrates, this attribute value is modified, and the migration timestamp is recorded. For example, if veh_10086 migrates from a warm layer to a hot layer, its "storage_layer" attribute is updated to "hot", and the corresponding migration timestamp is recorded.

[0150] Furthermore, update the time-series popularity index, including updating the real-time popularity entropy and storage layer identifier fields of the data entity to ensure accurate identification of the cache layer where the entity resides when querying by popularity. Update the relationship adjacency index, including updating the relationship strength value sorting of the entity in the relationship adjacency index if the access priority of the relationship changes due to entity migration. Update the attribute inverted index and subgraph structure fingerprint index, including updating only when the entity's attributes or structure change; if the attributes and structure remain unchanged during the migration process, only the storage layer information needs to be recorded.

[0151] The S3 also includes an access-driven graph heat update mechanism, specifically including:

[0152] S3.5: When a data entity in the cache is successfully accessed, update the real-time heat entropy of the data entity based on the timestamp of the access event;

[0153] In this embodiment, the base popularity value of access events is set according to the operation type: the base popularity value for query operations is 10, the base popularity value for modification operations is 8, and the base popularity value for related query operations is 5. For example, the current real-time popularity entropy of vehicle entity veh_10086 is 22.84. If it is queried once at the current time (1698765510000), its base popularity value is 10, Δt=0, and the updated popularity entropy = 22.84 + 10 × exp(-0.001 × 0) = 22.84 + 10 = 32.84; if it is modified once 5 seconds later (1698765515000), its base popularity value is 8, λ=0.001, Δt=5, and the updated popularity entropy = 8 × exp(-0.001 × 5) + 32.84 ≈ 7.96 + 32.84 = 40.8.

[0154] S3.6: Trigger related popularity propagation based on the operation type of the access request, including:

[0155] If the operation type is reading the attribute of the data entity, then the real-time heat entropy of the corresponding data entity is propagated to its directly connected attribute nodes. The propagated heat value is 10% of the entity's current heat entropy. The heat entropy update method of the attribute node is the same as that of the entity, that is, the updated heat entropy is equal to the product of the previous heat entropy and the first variable plus the current heat entropy. For example, when reading the veh_speed attribute of the entity veh_10086, the entity's current heat entropy is 40.8. The heat entropy propagated to the veh_speed attribute node is 40.8 × 0.1 = 4.08. The heat entropy of the attribute node = 40.8 + 4.08 × exp(-λ × Δt).

[0156] If the operation type is to traverse the relationship edges between the data entity and non-self data entities, then the real-time heat entropy of the corresponding data entity is propagated in a decaying manner along the traversed relationship edges to the associated data entities. The propagated heat value is the product of the entity's current heat entropy, the relationship strength value, and 0.5. The heat entropy update method of the associated data entities is the same as that of the original entity. For example, traversing the "through" relationship edge between entity veh_10086 and entity road_01, its relationship strength value is 0.74, the entity's current heat entropy is 40.8, and the heat value propagated to entity road_01 is 40.8 × 0.74 × 0.5 ≈ 15.09. The heat entropy of entity road_01 is equal to the original heat entropy plus the second variable, which is 15.09 × exp(-λ × Δt).

[0157] S3.7: Update the heat changes generated in steps S3.5 and S3.6 to the time-series heat index in real time to ensure that the index can reflect the latest heat status of the entity. In this embodiment, the heat update adopts an asynchronous update method, and the heat change event is processed in batches every 50ms. The latest heat entropy of the entity and attribute node is written to the Elasticsearch time-series heat index, and the update delay does not exceed 100ms;

[0158] S3.8: Re-evaluate the cache location of data entities based on the updated time-series popularity index;

[0159] If the real-time heat entropy change of any data entity causes the comprehensive score of its cache location decision function to cross the predefined hierarchical threshold, then the data entity is triggered to migrate between the hot, warm and cold layers.

[0160] For example, if the overall score of veh_10086 updates from 0.743 to 0.8105, crossing the hot layer threshold of 0.8, it triggers a migration from the warm layer to the hot layer. Similarly, if the overall score of any data source entity drops from 0.35 to 0.28, crossing the cold layer threshold of 0.3, it triggers a migration from the warm layer to the cold layer. The migration operation is scheduled and executed by Kubernetes, including three steps: data copying, data verification, and deletion of the original data. The entire migration process takes no more than 20ms, and the correctness of data access during the migration process is ensured through a cache consistency protocol.

[0161] Optionally, to address the limited capacity of the hot layer cache, a cache replacement strategy combining LRU and heat entropy is adopted. Specifically, when the hot layer cache capacity reaches 90%, cache replacement is triggered, prioritizing the replacement of entities that are "least recently used and whose real-time heat entropy is lower than the average heat entropy of the hot layer." For example, if there are 100 entities in the hot layer with an average heat entropy of 50, and entity A has not been used for the last 30 seconds with a heat entropy of 45 (lower than the average), while entity B has not been used for the last 10 seconds with a heat entropy of 55 (higher than the average), then entity A is prioritized for replacement. In this embodiment, compared to a simple LRU strategy, this optimization strategy improves the cache hit rate and reduces the access latency of hot data. LRU (least recently used) is a memory data eviction strategy, commonly used when memory is insufficient to evict the least recently used data. It is prior art in this field and not an inventive solution of this application, and will not be elaborated upon here.

[0162] Optionally, to address the issue of significant differences in system load across different time periods, the relationship strength threshold for preloading related data entities can be dynamically adjusted. Specifically, by monitoring system load, when the system load exceeds 80%, the preloading relationship strength threshold is increased from 0.6 to 0.7, reducing the number of preloaded entities and lowering system pressure; when the system load is below 30%, the preloading relationship strength threshold is decreased from 0.6 to 0.5, increasing the number of preloaded entities and improving subsequent access efficiency. In this embodiment, by dynamically adjusting the preloading relationship strength threshold, the system's response time during peak periods and cache hit rate during off-peak periods are improved.

[0163] Based on the association and confidence characteristics of each entity in the heterogeneous cache graph, labeled data is identified, data correction suggestions are generated and fed back to the incremental cleaning process to update the structured labeled data stream and the heterogeneous cache graph, including:

[0164] S4.1: Maintain a confidence trajectory for each attribute node in the heterogeneous cache graph; the confidence trajectory is used to record the attribute node identifier, historical values, value source, frequency of occurrence, last update timestamp, and value confidence. In this embodiment, the value confidence is calculated by weighting the reliability of the data source and the data consistency, wherein the reliability is represented by a weight. In this embodiment, the weight is 0.9 for the camera, 0.85 for the radar, 0.8 for the vehicle terminal, 0.95 for the traffic light controller, and 0.9 for the electronic police system.

[0165] For example, the confidence trajectory record for the veh_speed attribute node of vehicle entity veh_10086, identifying "7a3f9d...2b8e-veh_speed", is as follows:

[0166] Historical value: 65.3 km / h, value source: camera_01+radar_01+obd_01, frequency of occurrence: 120 times / minute, last update timestamp: 1698765438200, value confidence level: 0.93;

[0167] Historical value: 64.8 km / h, value source: camera_01+radar_01, frequency of occurrence: 80 times / minute, last update timestamp: 1698765408200, value confidence level: 0.89;

[0168] Historical value: 66.2 km / h, value source: obd_01, frequency of occurrence: 40 times / minute, last update timestamp: 1698765378200, value confidence level: 0.82;

[0169] Historical value: 35.7 km / h, source: radar_02, frequency of occurrence: 5 times / minute, last update timestamp: 1698765500000, confidence level: 0.78.

[0170] Furthermore, the update strategy for the confidence trajectory is as follows: whenever the attribute value of an attribute node changes, a new trajectory record is added; at the same time, trajectory records that are older than 3 days and appear less than 5 times / minute are deleted to avoid storage redundancy.

[0171] S4.2: Based on the heterogeneous cache graph, a predefined entity comprehensive confidence evaluation model is used to evaluate each data entity; the entity comprehensive confidence evaluation model aggregates the consistency of the confidence trajectory of all attribute nodes of the target data entity, the diversity of the sources of its association connection edges, and the stability characteristics of the corresponding data entity in historical correction, and outputs an entity confidence coefficient between 0 and 1.

[0172] In this embodiment, the entity comprehensive confidence assessment model is implemented using the TensorFlow framework. It is a multi-input single-output neural network model, and the input features include:

[0173] (1) Consistency of confidence trajectory of all attribute nodes of target data entity: Calculate the variance of the historical values ​​of each attribute node. The smaller the variance, the higher the consistency. Then take the average of the variances of all attribute nodes and normalize them to the [0,1] interval.

[0174] (2) Diversity of sources of association relationship edges: Count the number of data source types of association relationship edges and normalize them to the [0,1] interval. The more data source types there are, the larger the normalization value will be.

[0175] (3) Stability characteristics of data entities in historical corrections: Calculate the ratio of the number of historical corrections to the total number of data updates. The fewer the number of corrections, the higher the stability. Normalize to the [0,1] interval.

[0176] Furthermore, the network structure of the entity comprehensive confidence assessment model is as follows: the input layer has a dimension of 3, with each of the three input features occupying 1 dimension; the hidden layer contains two fully connected layers with 32 and 16 neurons respectively, and the activation function is Sigmoid; the output layer has a dimension of 1, and the output value is the entity confidence coefficient in the interval [0,1]. The closer the value is to 1, the more reliable the entity data is.

[0177] Furthermore, the model training process is as follows: In the offline stage, the model is trained using labeled data from the past three months, with Adam as the optimizer, a learning rate of 1e-3, 100 training epochs, and a binary cross-entropy loss function; In the online stage, the model is fine-tuned every 24 hours using the latest corrected data, with the learning rate adjusted to 5e-4. Adam and binary cross-entropy are existing technologies in this field and are not inventive solutions of this application, and will not be elaborated upon here.

[0178] For example, the input features of the vehicle entity veh_10086 are calculated as follows:

[0179] Consistency of confidence trajectory of attribute nodes: This entity has 5 core attribute nodes, including veh_speed, vehicle_type, license_plate, driving_direction, and position. The variances of the historical values ​​of each attribute node are 0.8, 0.0, 0.0, 0.5, and 1.2, respectively, and the average value is (0.8+0.0+0.0+0.5+1.2) / 5=0.5. The normalized value is 1-(0.5 / 5)=0.9. The maximum variance is preset to 5.

[0180] Diversity of connection sources: This entity has 3 connection edges, involving data source types of cameras, radar, and traffic incident systems, a total of 3 types, with a normalized value of 3 / 5 = 0.6. The maximum number of data source types is preset to 5.

[0181] Historical correction stability: In the past 100 data updates, this entity has been corrected 2 times, with a ratio of 2 / 100 = 0.02 and a normalized value of 1 - 0.02 = 0.98;

[0182] The three input features (0.9, 0.6, 0.98) are input into the entity comprehensive confidence assessment model. The output entity confidence coefficient is 0.92, which is higher than the confidence alarm threshold. In this embodiment, it is set to 0.7, and the entity is judged as a high-confidence entity.

[0183] S4.3: Periodically scan the heterogeneous cache map. When the entity confidence coefficient of any data entity is lower than the preset confidence alarm threshold, mark the data entity as a low-confidence entity and automatically generate a source tracing and correction proposal. The source tracing and correction proposal includes at least the conflicting attribute values, the comparison of values ​​from different sources, and the recommended correction values ​​based on statistical and correlation analysis.

[0184] In this embodiment, the preset confidence alarm threshold is 0.7, and the periodic scanning cycle is 1 minute. The specific process includes:

[0185] (1) A full scan is started every minute to traverse all data entities in Neo4j and call the entity comprehensive confidence evaluation model to calculate the entity confidence coefficient of each data entity;

[0186] (2) Mark entities with an entity confidence coefficient lower than 0.7 as low-confidence entities and record the marking timestamp and entity confidence coefficient;

[0187] (3) For low-confidence entities, analyze the confidence trajectory of all attribute nodes, identify conflicting attribute values, compare values ​​from different sources, and value frequency distribution. Among them, the attribute values ​​are those that differ significantly from most historical values.

[0188] (4) Based on statistical analysis and correlation analysis, such as combining the attribute values ​​of related entities, such as the speed limit information of road entities, a recommended correction value is generated.

[0189] For example, the entity confidence coefficient of pedestrian entity ped_005 is 0.65, which is lower than the confidence alarm threshold. The source analysis results are as follows:

[0190] Conflicting attribute values: The historical values ​​of the ped_age attribute are 25 years old and 65 years old, with a large difference between the two. The 25-year-old case comes from camera face recognition, which occurs 20 times / minute with a confidence level of 0.8. The 65-year-old case comes from infrared sensor, which occurs 5 times / minute with a confidence level of 0.7.

[0191] Association analysis: The road entity road_03 associated with this pedestrian entity has no special restrictions on the attribute "ped_crossing_age_limit", but the associated traffic event entity event_08 shows that the elderly in this area cross the street more frequently.

[0192] Recommended correction value: If the frequency of facial recognition data is higher, such as 20 times / minute or more than 5 times / minute and the confidence level is higher, the recommended correction value is 25 years old, and it is noted that "the infrared sensor data may contain misjudgments, and it is recommended to verify the device status".

[0193] S4.4: The generated source tracing correction proposal is encapsulated as a correction event and fed back to the incremental cleaning process. The incremental cleaning process updates the structured labeled data stream based on the correction event, thereby triggering the update of the heterogeneous cache graph.

[0194] Furthermore, the format of the correction event includes fields such as event identifier, entity identifier, correction attribute, recommended correction value, generation timestamp, and confidence level, which are sent to the incremental cleaning process via a message queue.

[0195] Furthermore, after receiving a correction event, the incremental cleaning process executes:

[0196] (1) Check whether the event signature is valid, whether the entity identifier exists, and whether the recommended correction value is within a reasonable range;

[0197] (2) Update the corresponding structured data record in the PostgreSQL database according to the recommended correction value in the correction event, and mark the correction timestamp and correction source;

[0198] (3) Notify the graph construction service through the CDC component to update the information of the corresponding entity and attribute nodes in Neo4j, including attribute values, confidence scores, and confidence scores.

[0199] CDC stands for Change Data Capture.

[0200] (4) Record the correction event, original data, and corrected data into the correction log library for model training and auditing.

[0201] For example, after receiving the correction event of ped_005, update its ped_age attribute value from 65 to 25, update the confidence level from 0.78 to 0.85, synchronously update the information of the attribute node in the heterogeneous cache graph, and record the correction log.

[0202] In step S4, corresponding subgraph replicas of the heterogeneous cached graph are maintained locally on each data source node. The graph neural network model parameters and hotspot entity states of each subgraph replica are synchronized using a secure aggregation algorithm, including:

[0203] S4.5: Each participating data source node, based on its data jurisdiction, cuts out a corresponding subgraph copy from the global dynamic entity association graph for maintenance; the subgraph copy is a slice of the heterogeneous cache graph;

[0204] Furthermore, the scope of data jurisdiction is divided according to physical area and data type: for example, the jurisdiction of the roadside camera camera_01 is "lane 1 from XX intersection to YY intersection", and the data type is image annotation data of vehicles and pedestrians. Therefore, its subgraph copy only contains vehicle and pedestrian entities in this area, as well as image-related attribute nodes of these entities, associated road entities, and related relational edges.

[0205] Furthermore, the subgraph replicas are stored using a lightweight graph database, occupying approximately 15% of the storage space of the global graph, and supporting fast local queries and updates. Each data source node deploys a slice synchronization service to receive change events from the global graph in real time, update the subgraph replicas, and ensure the consistency between the subgraph replicas and the global graph, i.e., the synchronization latency does not exceed 50ms.

[0206] S4.6: Establish a synchronization cycle. Within each cycle, each data source node executes:

[0207] The time-series graph neural network is trained using local data, and the gradients of the model parameters calculated after training are homomorphically encrypted to form encrypted gradient data. Homomorphic encryption is a prior art in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0208] Extract the top K data entities with the highest real-time heat entropy from the subgraph replica, obtain the entity confidence coefficient and attributes of the data entities, form a local hot entity state snapshot, and encrypt the local hot entity state snapshot;

[0209] Furthermore, the local hotspot entity status snapshot extraction process includes: based on the time-series heat index of the subgraph replica, extracting the top K data entities with the highest real-time heat entropy (K=50 in this embodiment), obtaining the entity fingerprint, entity confidence coefficient, and core attribute value of these data entities, and forming a local hotspot entity status snapshot.

[0210] Furthermore, the AES-256 encryption algorithm is used to encrypt the local hotspot entity state snapshot. The AES key is distributed to each data source node by the central coordinator through a secure channel. Each node's key is independent and updated regularly. The AES-256 encryption algorithm is existing technology in this field and is not an inventive solution of this application, so it will not be described in detail here.

[0211] The encrypted gradient data and the encrypted snapshot of the local hotspot entity status are uploaded to the central coordinator.

[0212] The S4 step, which involves maintaining corresponding subgraph replicas of the heterogeneous cached graph locally on each data source node and synchronizing the graph neural network model parameters and hotspot entity states of each subgraph replica using a secure aggregation algorithm, also includes:

[0213] S4.7: After receiving the encrypted information uploaded by all data source nodes, the central coordinator executes a secure aggregation algorithm, including: performing a secure average calculation on the encrypted gradient data from all data source nodes, decrypting the calculation result, and using the decrypted calculation result to update the graph neural network model, which is a temporal graph neural network; simultaneously, merging the encrypted local hotspot entity state snapshots from all data source nodes, analyzing to obtain the global hotspot and confidence distribution status; and distributing the updated graph neural network model parameters to all data source nodes.

[0214] Furthermore, the corresponding AES key is used to decrypt the encrypted local hotspot entity status snapshots uploaded by all data source nodes, extracting information such as entity fingerprint code, entity confidence coefficient, and core attribute value from each snapshot, and deduplicating and merging them according to the entity fingerprint code: for the same data entity, the average of the entity confidence coefficient is taken as the global confidence coefficient, and the mode of the core attribute value is taken as the global attribute value. The heat distribution of the data entity on each node is recorded to form a global hotspot and confidence distribution status report.

[0215] S4.8: Each data source node receives the graph neural network model parameters issued by the central coordinator, updates its local time-series graph neural network, and adjusts the state of relevant data entities in the subgraph replicas.

[0216] Furthermore, each data source node receives the global model parameters issued by the central coordinator and completes the local update according to the following steps:

[0217] (1) Verify the RSA signature of the data to be sent to ensure that the data has not been tampered with and comes from a legitimate central coordinator; if the signature verification fails, refuse to update and send an anomaly report to the central coordinator.

[0218] (2) Replace the corresponding parameters of the local temporal graph neural network with the subset of graph neural network model parameters issued. After the update, perform a local data inference test. If the inference accuracy drops by more than 3% compared with before the update, roll back to the local model parameters before the update and record the model update anomaly; otherwise, confirm that the model update is effective.

[0219] (3) Based on the global hotspot and confidence distribution status report, adjust the status of data entities in the subgraph replica: For the global Top 100 hotspot entities, increase their weight in the local time-series heat index and prioritize the allocation of local cache resources; For entities in the global low-confidence entity list, mark them as pending review status and trigger local secondary confidence assessment; For entities whose core attribute values ​​differ from global attribute values ​​by more than 5%, recalculate the entity confidence coefficient in combination with the local confidence trajectory and generate local correction suggestions when necessary.

[0220] For example, after receiving the global model parameters, the data source node camera_01 updates the graph attention layer parameters of the local temporal graph neural network, improving the inference test accuracy by 2.5%, confirming the effectiveness of the update, and marking veh_10086 as a global hot entity, increasing its local cache priority; it was found that the ped_age attribute value (65 years old) of the local entity ped_005 differed significantly from the global attribute value (25 years old), and its entity confidence coefficient was recalculated to 0.63, marking it as pending review.

[0221] The specific steps of S5 include:

[0222] S5.1: When an access request for a target data entity is received, the subject identity, operation type, and environment token are extracted from the context of the access request;

[0223] Furthermore, regarding the identity of the subject: the subject identity is verified through an OAuth2.0 token, and the subject type in the token is parsed, such as traffic management department / third-party data analyst / ordinary citizen, as well as the subject ID and authorized permission list, where the authorized permission list includes, for example, querying vehicle speed, modifying traffic event status, and read-only access to road information.

[0224] Furthermore, regarding operation types: clearly define the type of operation requested, including four categories: query, modification, deletion, and related queries. Different operation types correspond to different access control rules.

[0225] Furthermore, for environment tokens: extract the environment information of the request, including the access IP address, device type, access time, and network type, for dynamic risk assessment. For example, additional verification is required for access to sensitive entities from the external network.

[0226] For example, a third-party data analyst, analyzer_003, initiates an access request through an external PC to query the veh_speed attribute of entity veh_10086. The API gateway resolves the subject's identity as medium-level, the operation type as query, and the environment token as "IP:203.xxx.xxx.xxx, Device type: PC, Access time: Weekday 14:30, Network type: External".

[0227] S5.2: Query the heterogeneous cache graph to obtain all attribute nodes of the target data entity, associated data entities and their relationship types, and dynamically generate the context feature vector of this access by combining the historical access records of the subject identity;

[0228] In this embodiment, the context feature vector is 12-dimensional, including:

[0229] (1) Sensitivity level of target entity: Based on the entity attributes and association relationship, it is divided into three levels: high, medium and low, which are quantified as 1.0, 0.5 and 0.0 respectively. Among them, the high level is such as accident injury and death information and pedestrian identity characteristics in traffic incidents, the medium level is such as vehicle speed and road congestion level, and the low level is such as road segment number and number of lanes.

[0230] (2) Subject authority level: high authority, such as traffic management departments, is quantified as 1.0; medium authority, such as third-party data analysts, is quantified as 0.6; low authority, such as ordinary citizens, is quantified as 0.3.

[0231] (3) Operation type risk coefficient: query is 0.2, related query is 0.4, modification is 0.7, and deletion is 1.0;

[0232] (4) Environmental risk coefficient: The risk coefficient for intranet access is quantified as 0.1, and the risk coefficient for extranet access is quantified as 0.5; the risk coefficient for access during working hours is quantified as 0.2, and the risk coefficient for access during non-working hours is quantified as 0.4; the risk coefficient for server type is quantified as 0.1, the risk coefficient for PC is quantified as 0.3, and the risk coefficient for mobile terminal is quantified as 0.5; the environmental risk coefficient is equal to the weighted sum of the risk coefficients of each sub-item, and the weight of each sub-item is 0.25;

[0233] (5) Subject's historical access compliance rate: The proportion of the number of access requests of the subject that have passed compliance in the past 30 days to the total number of requests. For example, analyze_003 has 100 accesses in the past 30 days, all of which have been compliant, and the compliance rate is 1.0.

[0234] (6) The degree of relevance between the subject and the target entity: If the subject's jurisdiction / authorization scope is consistent with the target entity's physical location / data ownership, the value is 0.8; otherwise, it is 0.2.

[0235] (7) Real-time heat entropy of the target entity: taken from the latest value in the time-series heat index and normalized to the [0,1] interval;

[0236] (8) Confidence coefficient of the target entity: taken from the entity confidence coefficient in the heterogeneous cache graph and normalized to the [0,1] interval;

[0237] (9) Subject historical access frequency: The average number of times the subject accesses the target entity's entity type in the past 7 days, normalized to the [0,1] interval. For example, analyze_003 accesses the vehicle entity an average of 5 times per day in the past 7 days, with a normalized value of 0.5.

[0238] (10) Cache layer location of the target entity: hot layer quantization is 0.8, warm layer quantization is 0.5, cold layer quantization is 0.2;

[0239] (11) The urgency of the access request: determined by the "emergency" field in the request header, with urgency quantified as 1.0 and non-urgency quantified as 0.0;

[0240] (12) Number of historical conflict accesses: The number of times the subject was rejected or downgraded when accessing the same entity or similar entities in the past 30 days, normalized to the range of [0,1]. If there is no conflict access, it is quantified as 0.0.

[0241] S5.3: Input the generated context feature vector into the deep policy evaluation network and output an arbitration decision; the arbitration decision is one of allow, deny, or downgrade, wherein the downgrade decision means that only the de-identified simplified version of the target data entity stored in the cold layer is allowed to access;

[0242] The deep policy evaluation network is implemented using the TensorFlow framework and is used to output access arbitration decisions. The network structure and training process are as follows:

[0243] (1) Network structure: The input layer has a dimension of 12, which is the context feature vector. The hidden layer contains 3 fully connected layers with 64, 32 and 16 neurons respectively. The activation function is LeakyReLU. The output layer uses the Softmax activation function and outputs 3 probability values, which correspond to the three decisions of allow, reject and downgrade respectively. The one with the highest probability value is the final arbitration decision.

[0244] (2) Training process: In the offline stage, the network is trained using access log data from the past 6 months. The optimizer is Adamax, the learning rate is 1e-3, the training rounds are 200, and the loss function is cross-entropy loss. In the online stage, the network is fine-tuned every 12 hours using the latest access logs and manual review results to ensure that the network adapts to the dynamic changes in access patterns.

[0245] (3) Decision definition:

[0246] Allow: Grant the subject full access rights to read / modify / delete the target entity's raw data, i.e., thermal / thermal data;

[0247] Deny: Prevents the subject from accessing the target entity and returns a message indicating "Insufficient permissions or high access risk";

[0248] Degradation: Only allow the subject to access the simplified version of the target entity stored in the cold layer after desensitization, such as hiding the last 4 digits of the license plate, retaining the integer part of the speed, and anonymizing identity features.

[0249] For example, inputting the aforementioned 12-dimensional context feature vector into a deep policy evaluation network yields the following probabilities: Allow 0.75, Deny 0.10, Downgrade 0.15, with the final arbitration decision being "Allow". If the subject is an ordinary citizen user_12345 accessing the license_plate attribute of veh_10086, the generated feature vector has a sensitivity level of 1.0 and a permission level of 0.3, resulting in a network output denial probability of 0.85 and a decision of "Deny". If accessing the veh_speed attribute of veh_10086, the network outputs a downgrade probability of 0.60, resulting in a decision of "Downgrade", allowing access only to the desensitized speed value in the cold layer.

[0250] S5.4: Enforcing the arbitration decision, including:

[0251] If the arbitration decision permits, then access rights shall be granted;

[0252] If the arbitration decision is to downgrade or reject, the access event is first recorded, and then a cache downgrade instruction for the target data entity is sent to the hierarchical caching architecture. The cache downgrade instruction is used to force the value of the target data entity to zero when updating the real-time heat entropy.

[0253] Furthermore, the decision is "allow": The API gateway sends an access request to the hierarchical caching architecture, obtains the original data of the target data entity in the corresponding caching layer, and returns it to the accessing subject; at the same time, it records the access log, including the subject's identity, operation type, access time, entity information, and decision result.

[0254] Furthermore, the decision is "reject": the API gateway directly returns a response denying access and does not send access requests to the tiered caching architecture; it records access logs and reasons for rejection, such as "insufficient permissions" or "too high sensitivity level". If the same subject is denied access to the same type of entity three times in a row, it triggers secondary identity verification, such as SMS verification code or face verification.

[0255] Furthermore, the decision is to "downgrade": the API gateway sends an access request to the cold layer cache, obtains the simplified version of the data after de-identification, and returns it to the access subject; it records the access log and the downgrade flag, and marks the number of times the entity has accessed the de-identified data, which is used for caching strategy adjustment.

[0256] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of the present invention without departing from the spirit and scope of the present invention. All of these variations are within the protection scope of the present invention.

Claims

1. A method for intelligent caching of real-time labeled data streams, characterized in that, include: S1: Receives multi-source labeled data streams, maps labeled fields from different sources to nodes of a temporal graph neural network, calculates semantic and structural similarity between nodes, generates field mapping rules and performs incremental cleaning, and outputs structured labeled data streams. S2: Using the structured labeled data stream as input, extract the data entities, entity attributes, and relationships between entities contained therein, construct a heterogeneous cache graph, and establish a multi-dimensional index; S3: Based on the heterogeneous cache graph, construct a hierarchical cache architecture that includes a hot layer, a warm layer and a cold layer, dynamically allocate data entities in the heterogeneous cache graph to the corresponding cache layer, and synchronously update the heterogeneous cache graph status and multi-dimensional index in real time. S4: Based on the association and confidence characteristics of each data entity in the heterogeneous cache graph, identify the labeled data, generate data correction suggestions and feed them back to the incremental cleaning process to update the structured labeled data stream and the heterogeneous cache graph. At the same time, maintain the corresponding subgraph copy of the heterogeneous cache graph locally on each data source node, and synchronize the graph neural network model parameters and hot spot entity status of each subgraph copy through a secure aggregation algorithm. S5: Based on the attributes and association characteristics of data entities in the heterogeneous cache graph, perform dynamic access control on the data stored in the hierarchical cache architecture, and feed the control decision back to S3.

2. The intelligent caching processing method for real-time labeled data streams as described in claim 1, characterized in that, S1 specifically includes: S1.1: Create a temporal feature node for each labeled field from each source; the attributes of the temporal feature node include the semantic embedding vector of the field name, the statistical feature vector of the value range distribution, and the timestamp sequence of field occurrence and change events; S1.2: Input all temporal feature nodes within the continuous time window into the temporal graph neural network. The temporal graph neural network captures the temporal dependency features of each temporal feature node through the time-gated recurrent unit layer, and then aggregates the cross-source structural features between temporal feature nodes from different data sources through the graph attention layer, and outputs the fused feature vector of each temporal feature node. S1.3: Calculate the cross-source semantic affinity between the fused feature vectors of any two temporal feature nodes, and calculate the temporal co-occurrence intensity based on the co-occurrence relationship of the two temporal feature nodes within the sliding time window. Then, input the cross-source semantic affinity and the temporal co-occurrence intensity into the rule generation neural network. The rule generation neural network outputs a probabilistic field mapping rule matrix, where each element in the field mapping rule matrix represents the confidence weight of mapping one source field to another source field. S1.4: Based on the field mapping rule matrix, the original labeled fields that flow in in real time are parsed and aligned. Data with a mapping confidence level higher than the preset first threshold are directly fused. Data with a confidence level lower than the preset first threshold but higher than the preset second threshold are marked and temporarily stored in the pending area. Data with a confidence level lower than the preset second threshold is triggered with an anomaly alarm. Finally, a structured labeled data stream is output.

3. The intelligent caching processing method for real-time labeled data streams as described in claim 1, characterized in that, The process of constructing the heterogeneous cache map in S2 includes: S2.1: Identify data entities from the structured labeled data stream and assign a globally unique entity fingerprint code to each data entity; the entity fingerprint code is generated by combining its entity type hash value and attribute hash value; S2.2: Extract the attribute key-value pairs of each data entity, instantiate each attribute key-value pair into an attribute node, and link each attribute node to its data entity through an edge; S2.3: Identify the relationships between different data entities, instantiate each relationship as a directed relationship edge, and assign the relationship edge a relationship type, establishment time, and a relationship strength value based on co-occurrence frequency and semantic consistency; S2.4: Construct a heterogeneous cache graph based on all identified data entities, all generated attribute nodes and their links to data entities, and all generated relational links. The heterogeneous cache graph is composed of data entities as vertices, attribute nodes as subordinate vertices, and links between data entities and attribute nodes and relational links between data entities as edges.

4. The intelligent caching processing method for real-time labeled data streams as described in claim 3, characterized in that, The multi-dimensional index in S2 is a four-element composite index structure, which includes an attribute inverted index, a relation adjacency index, a time-series heat index, and a subgraph structure fingerprint index. The attribute inverted index is an inverted list built for the keys and values ​​of all attribute nodes, used for fuzzy matching and range queries based on attribute values; The relation adjacency index is used to record the relation type, target data entity, and current relation strength value of all relation connection edges for each data entity, and is sorted in descending order of relation strength value; The time-series heat index is based on the event timestamps of when a data entity is accessed, associated, or its attributes are modified. It uses a time decay function to dynamically calculate the real-time heat entropy of each data entity and establishes a sorted index of the real-time heat entropy. The subgraph structure fingerprint index calculates the local subgraph structure within two hops centered on each data entity, generating a fixed-length subgraph structure fingerprint for graph-based similarity retrieval.

5. The intelligent caching processing method for real-time labeled data streams as described in claim 4, characterized in that, The specific steps of S3 include: S3.1: Divide the cache storage medium into three logical layers, including a hot layer, a warm layer, and a cold layer; S3.2: Based on the heterogeneous cache map, a cache location decision function dynamically allocates data entities in the heterogeneous cache map to the divided cache layers; the cache location decision function uses the real-time heat entropy of the target data entity, the average relationship strength value of its associated edges, and the storage overhead of the target data entity as input parameters to calculate a comprehensive score; according to the predefined numerical range of the comprehensive score, the target data entity is allocated to the hot layer, warm layer, or cold layer; S3.3: When any data entity is migrated to the hot layer according to the decision, based on the relational adjacency index, retrieve the associated data entities that have relational connection edges with the data entities in the hot layer and are currently located in the warm or cold layer, and preload the associated data entities into the preparation buffer of the warm layer. S3.4: After the data entity migrates between cache layers according to S3.2-S3.3, the storage location status of the corresponding data entity in the heterogeneous cache graph is updated synchronously, and the entries associated with the corresponding data entity in the multi-dimensional index are updated.

6. The intelligent caching processing method for real-time labeled data streams as described in claim 5, characterized in that, The S3 also includes an access-driven graph heat update mechanism, specifically including: S3.5: When a data entity in the cache is successfully accessed, update the real-time heat entropy of the corresponding data entity based on the timestamp of the access event; S3.6: Trigger related popularity propagation based on the operation type of the access request, including: If the operation type is to read the attribute of the data entity, then the real-time heat entropy of the corresponding data entity is propagated to the attribute node directly connected to it. If the operation type is to traverse the relationship edges between the data entity and non-self data entities, then the real-time heat entropy of the corresponding data entity is propagated in a decaying manner along the traversed relationship edges to the associated data entities. S3.7: Update the heat changes generated in steps S3.5 and S3.6 to the time-series heat index in real time; S3.8: Re-evaluate the cache location of data entities based on the updated time-series popularity index; If the real-time heat entropy change of any data entity causes the comprehensive score of its cache location decision function to cross the predefined hierarchical threshold, then the corresponding data entity is triggered to migrate between the hot layer, the warm layer, and the cold layer.

7. The intelligent caching processing method for real-time labeled data streams as described in claim 1, characterized in that, Based on the association and confidence characteristics of each entity in the heterogeneous cache graph, labeled data is identified, data correction suggestions are generated and fed back to the incremental cleaning process to update the structured labeled data stream and the heterogeneous cache graph, including: S4.1: In the heterogeneous cache graph, maintain a confidence trajectory for each attribute node; the confidence trajectory is used to record the attribute node identifier, historical values, value source, frequency of occurrence, last update timestamp, and value confidence. S4.2: Based on the heterogeneous cache graph, a predefined entity comprehensive confidence evaluation model is used to evaluate each data entity; the entity comprehensive confidence evaluation model aggregates the consistency of the confidence trajectory of all attribute nodes of the target data entity, the diversity of the sources of its association connection edges, and the stability characteristics of the corresponding data entity in historical correction, and outputs an entity confidence coefficient between 0 and 1. S4.3: Periodically scan the heterogeneous cache map. When the entity confidence coefficient of any data entity is lower than the preset confidence alarm threshold, mark the corresponding data entity as a low-confidence entity and automatically generate a source tracing and correction proposal. The source tracing and correction proposal includes at least the conflicting attribute values, the comparison of values ​​from different sources, and the recommended correction values ​​based on statistical and correlation analysis. S4.4: The generated source tracing correction proposal is encapsulated as a correction event and fed back to the incremental cleaning process. The incremental cleaning process updates the structured labeled data stream based on the correction event, triggering the update of the heterogeneous cache map.

8. The intelligent caching processing method for real-time labeled data streams as described in claim 1, characterized in that, In step S4, corresponding subgraph replicas of the heterogeneous cached graph are maintained locally on each data source node. The graph neural network model parameters and hotspot entity states of each subgraph replica are synchronized using a secure aggregation algorithm, including: S4.5: Each participating data source node, based on its data jurisdiction, cuts out a corresponding subgraph copy from the global dynamic entity association graph for maintenance; the subgraph copy is a slice of the heterogeneous cache graph; S4.6: Establish a synchronization cycle. Within each cycle, each data source node executes: The time-series graph neural network is trained using local data, and the gradients of the model parameters calculated after training are homomorphically encrypted to form encrypted gradient data. Extract the top K data entities with the highest real-time heat entropy from the subgraph replica, obtain the entity confidence coefficient and attributes of the data entities, form a local hot entity state snapshot, and encrypt the local hot entity state snapshot; The encrypted gradient data and the encrypted snapshot of the local hotspot entity status are uploaded to the central coordinator.

9. The intelligent caching processing method for real-time labeled data streams as described in claim 8, characterized in that, The S4 step, which involves maintaining corresponding subgraph replicas of the heterogeneous cached graph locally on each data source node and synchronizing the graph neural network model parameters and hotspot entity states of each subgraph replica using a secure aggregation algorithm, also includes: S4.7: After receiving the encrypted information uploaded by all data source nodes, the central coordinator executes a secure aggregation algorithm, including: performing a secure average calculation on the encrypted gradient data from all data source nodes, decrypting the calculation result, and using the decrypted calculation result to update the graph neural network model, which is a temporal graph neural network; simultaneously, merging the encrypted local hotspot entity state snapshots from all data source nodes, analyzing to obtain the global hotspot and confidence distribution status; and distributing the updated graph neural network model parameters to all data source nodes. S4.8: Each data source node receives the graph neural network model parameters issued by the central coordinator, updates its local time-series graph neural network, and adjusts the state of relevant data entities in the subgraph replicas.

10. The intelligent caching processing method for real-time labeled data streams as described in claim 1, characterized in that, The specific steps of S5 include: S5.1: When an access request for a target data entity is received, the subject identity, operation type, and environment token are extracted from the context of the access request; S5.2: Query the heterogeneous cache graph to obtain all attribute nodes of the target data entity, associated data entities and their relationship types, and dynamically generate the context feature vector of this access by combining the historical access records of the subject identity; S5.3: Input the generated context feature vector into the deep policy evaluation network and output an arbitration decision; the arbitration decision is one of allow, deny, or downgrade, wherein the downgrade decision means that only the de-identified simplified version of the target data entity stored in the cold layer is allowed to access; S5.4: Enforcing the arbitration decision, including: If the arbitration decision permits, then access rights shall be granted; If the arbitration decision is to downgrade or reject, the access event is first recorded, and then a cache downgrade instruction for the target data entity is sent to the hierarchical caching architecture. The cache downgrade instruction is used to force the value of the target data entity to zero when updating the real-time heat entropy.

Images