Cold and hot data management method and system and server

Abstract

The invention relates to the technical field of servers, and discloses a cold and hot data management method and system and a server. According to the method, data access behavior information is obtained, multi-dimensional prediction and business rule collaborative decision are combined, a data storage scheduling strategy is generated, and corresponding storage resource allocation operation is executed. According to the invention, the utilization efficiency of storage resources can be improved, the response capability of the system to high-value data is enhanced, and the compliance and safety requirements under the scenes of finance, e-commerce and the like are met. The method can be widely applied to cloud computing, big data platforms and distributed storage systems.

Description

Technical Field

[0001] This application relates to the field of server technology, specifically to a method, system, and server for managing hot and cold data. Background Technology

[0002] In large-scale data processing systems hosted on servers, data access exhibits a significant "hot / cold" imbalance. Identifying the hot / cold status of data is fundamental for achieving tiered management of storage resources and improving system service performance. Especially in high-concurrency scenarios such as finance, e-commerce, and cloud computing, the rationality of data scheduling strategies directly impacts hardware resource utilization and user experience.

[0003] Existing methods for identifying hot and cold data are mostly based on simple statistical information such as access frequency or timestamps, which have limited adaptability to complex access patterns and often lead to resource scheduling delays or mismatches. In the management of sensitive data involving auditing and privacy, automated scheduling mechanisms lack the necessary intervention capabilities, potentially posing compliance risks. Furthermore, if intermediate data generated during processing is not properly handled, there is a risk of information residue.

[0004] Therefore, improving the adaptability of hot and cold data identification in server environments and meeting the management requirements of complex business environments has become a technical challenge in this field. Summary of the Invention

[0005] This application provides a method, system, and server for managing hot and cold data, in order to solve the problems of low accuracy in identifying hot and cold data, difficulty in responding to business rule requirements, and lack of security guarantees in intermediate data processing in the prior art.

[0006] At the method level, the method includes: acquiring data access events and generating behavioral information to characterize data access behavior based on the data access events; generating a first cold / hot determination result and a second cold / hot determination result based on the behavioral information; performing a fusion decision on the first cold / hot determination result and the second cold / hot determination result, and generating a final storage strategy in combination with preset business rules; and executing data storage scheduling operations according to the final storage strategy.

[0007] Furthermore, the generation of the first hot / cold determination result includes:

[0008] Each data item is treated as a node. If any two data items are accessed consecutively by the same object within a preset time window, a connection is established between the corresponding nodes. The weight of the connection is the number of consecutive accesses.

[0009] Aggregate the associated node information of each node, determine the degree of influence of each associated node on the current node through a preset association strength evaluation method, and obtain the node fusion information by weighted summation;

[0010] The original node information is combined with the fused information to generate a popularity score, which serves as the first hot / cold determination result.

[0011] Furthermore, the generation of the second hot / cold determination result includes:

[0012] Predict future access behavior based on the historical access sequence of data items. The historical access sequence includes multiple access features arranged in chronological order and is truncated to form an input sequence using a sliding window method.

[0013] The prediction results are converted into heat labels as the second hot / cold determination result.

[0014] Furthermore, the step of generating the final storage strategy by combining preset business rules includes:

[0015] Prioritize the execution of preset business rules, including assigning the highest priority to compliance rules, followed by security rules, and the lowest priority to performance rules;

[0016] When generating the strategy, rules are matched according to priority identifiers. Compliance rules include mandating that financial audit data be stored on low-speed storage media, or that high-value unpaid orders be cached on high-speed storage media.

[0017] Furthermore, when preset compliance trigger conditions are met, a manual intervention process is initiated. These preset compliance trigger conditions include data items carrying financial audit, privacy protection, or anti-money laundering labels.

[0018] Once the manual intervention process is initiated, an audit log is generated containing the identity of the auditor, the audit duration, and the basis for the decision. The key hash value of the audit log is then submitted to the blockchain for immutable evidence storage.

[0019] At the system level, the system includes a feature generation module, a parallel prediction module, a rule-based decision-making module, and a storage scheduling module. The feature generation module is used to acquire data access events and generate behavioral information characterizing data access behavior based on these events.

[0020] The parallel prediction module is used to generate a first cold / hot determination result and a second cold / hot determination result based on the behavioral information;

[0021] The rule decision module is used to receive the first cold / hot determination result and the second cold / hot determination result, perform a fusion decision on the two, and generate the final storage strategy in combination with preset business rules;

[0022] The storage scheduling module is used to perform data storage scheduling operations according to the final storage strategy.

[0023] The parallel prediction module includes a first prediction channel and a second prediction channel. The first prediction channel is used to generate a popularity score based on the access correlation between data items.

[0024] The popularity score is a numerical value that represents the likelihood of a data item being accessed in the future.

[0025] Furthermore, the first prediction channel generates a popularity score in the following manner:

[0026] Each data item is treated as a node. If any two data items are accessed consecutively by the same object within a preset time window, a connection is established between the corresponding nodes. The weight of the connection is the number of consecutive accesses.

[0027] Aggregate the associated node information of each node, determine the degree of influence of each associated node on the current node through a preset association strength evaluation method, and obtain the node fusion information by weighted summation;

[0028] The original node information is combined with the fused information to generate a popularity score.

[0029] Furthermore, the second prediction channel is used to predict future access behavior of data items based on their historical access sequences;

[0030] The historical access sequence includes multiple access features arranged in chronological order, and is extracted to form the input sequence using a sliding window method.

[0031] Furthermore, the access prediction results output by the second prediction channel are converted into heat labels as the second hot / cold determination results, which are used to guide the admission strategy of Adaptive Replacement Cache (ARC). Data items predicted as "hot" skip the L1 cache admission evaluation process of ARC and are directly written to the head position of the T2 queue.

[0032] Furthermore, the preset business rules include mandating that financial audit data be stored on low-speed storage media, or mandating that high-value unpaid orders be cached on high-speed storage media;

[0033] The rule decision module prioritizes preset business rules, including assigning the highest priority identifier to compliance rules, followed by security rules, and the lowest priority to performance rules, and matching rules according to the priority identifier order when generating the policy.

[0034] Furthermore, the rule decision module is also used to initiate a manual intervention process when preset compliance trigger conditions are met;

[0035] The preset compliance trigger conditions include data items carrying labels related to financial auditing, privacy protection, or anti-money laundering.

[0036] Furthermore, the manual intervention process includes generating audit logs containing the identity of the auditors, the audit duration, and the basis for the decisions.

[0037] Specifically, the key hash value of the audit log is submitted to the blockchain for immutable evidence storage.

[0038] Furthermore, the method also includes a step of securely destroying intermediate processing data: when the predictive processing is completed and the intermediate data is not referenced, when the system detects abnormal access behavior, or when the manual intervention process is initiated, a secure destruction process is triggered, a data destruction certificate is generated, and the certificate hash is submitted to the blockchain for evidence storage.

[0039] Furthermore, the system also includes a secure destruction module, used to securely destroy intermediate processed data when certain conditions are met, generate a data destruction certificate, and submit the certificate hash to the blockchain for notarization, ensuring the auditability and security of the data processing process. Specifically, the generated data destruction certificate includes the fields to be destroyed, the witness, the destruction method, and the certificate hash, and the certificate hash is stored on the blockchain for notarization.

[0040] Furthermore, the node information used by the first prediction channel in generating the popularity score includes:

[0041] Historical access frequency of data items;

[0042] Average access time for data items;

[0043] The last access timestamp of the data item;

[0044] Metadata type identifier for data items.

[0045] Furthermore, the secure destruction module initiates the destruction process when any of the following conditions are met:

[0046] Prediction processing is complete and intermediate data is not referenced by subsequent tasks;

[0047] The system detected abnormal access behavior;

[0048] The manual intervention process has been initiated.

[0049] Furthermore, when the first and second cold / hot determination results are inconsistent, the rule decision module initiates a conflict arbitration mechanism, including:

[0050] Retrieve the business attribute tags of data items;

[0051] If the business attribute tag belongs to a preset high priority category, then the first hot / cold determination result is adopted;

[0052] Otherwise, the second cold / hot determination result shall be adopted.

[0053] At the server level, the server includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the program to implement the steps of the above method.

[0054] The above solutions can improve the accuracy of hot and cold data identification, enhance the flexibility of storage resource scheduling, and meet the management requirements of business compliance and data security. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the overall process of a cold and hot data management method provided in one embodiment;

[0056] Figure 2 This is a schematic diagram of the process for generating a first hot / cold determination result based on the correlation strength assessment of a graph structure, provided in one embodiment.

[0057] Figure 3 This is a schematic diagram of a scheme for generating a second cold / hot determination result based on time series prediction, provided in one embodiment.

[0058] Figure 4 This is a flowchart illustrating a priority-based business rule fusion mechanism scheme provided in one embodiment;

[0059] Figure 5 This is a schematic diagram of the framework of a cold and hot data management system provided in one embodiment;

[0060] Figure 6 This is a schematic diagram of the server structure provided in one embodiment. Detailed Implementation

[0061] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the described embodiments are only for explaining the present invention and not for limiting the present invention. Without departing from the spirit and scope of the present invention, those skilled in the art can make appropriate modifications or equivalent substitutions to the present invention, all of which should be included within the protection scope of the present invention.

[0062] In one embodiment, a cold and hot data management method is provided to address the problems of poor adaptability to complex access patterns, delayed scheduling response, lack of compliance intervention mechanisms, and information residue risks in intermediate data processing in existing cold and hot data identification methods, and to dynamically manage the storage status of large-scale data in a server environment.

[0063] like Figure 1 As shown, the overall process of this hot and cold data management method includes:

[0064] S101: Obtain data access events and generate behavioral information to characterize data access behavior based on the data access events; the behavioral information includes at least one of the following: access subject identifier, accessed data item identifier, access timestamp, access type, and access duration.

[0065] S102: Generate a first and a second hot / cold rating result based on behavioral information. Specifically, based on the behavioral information, a first and a second hot / cold rating result are generated respectively through a first prediction channel and a second prediction channel executed in parallel. The first hot / cold rating result is a popularity score generated based on the access correlation between data items, and the second hot / cold rating result is a popularity tag generated based on historical access sequence prediction.

[0066] S103: The first and second cold / hot determination results are merged for decision-making, and a final storage strategy is generated by combining them with preset business rules. Specifically, the preset business rules are prioritized according to compliance, security, and performance categories, and are matched according to priority order during strategy generation.

[0067] S104: Execute data storage scheduling operations according to the final storage policy.

[0068] Data access events originate from system logs, database access records, or cache monitoring modules. Behavioral information includes the accessing entity identifier, the accessed data item identifier, the access timestamp, the access type, and the access duration. The first hot / cold determination result is a popularity score generated based on the access relationships between data items; the second hot / cold determination result is a popularity tag generated based on historical access sequence predictions. The fusion decision process receives both determination results as input and outputs a fused intermediate determination result. Pre-set business rules are stored in a rule base, and the rule decision module matches rules according to priority. Data storage scheduling operations include migrating data between high-speed and low-speed storage media, cache loading, cache eviction, or archiving to cold storage.

[0069] Furthermore, in one embodiment, an architecture design scheme of "dual-channel prediction + rule fusion decision" is also included (not shown in the figure). The first and second cold / hot determination results reflect the characteristics of data access behavior in the two dimensions of association path and time series, respectively. These two results are generated in parallel and complement each other. The rule decision module receives the two determination results as input, performs fusion decision, and generates the final storage strategy by combining preset business rules. This architecture achieves the synergy of predictive judgment based on behavior analysis and rule-based control based on business constraints, forming a hybrid scheduling mechanism that balances adaptability and controllability.

[0070] This method generates two cold / hot determination results in parallel, which can reflect the data access characteristics from different dimensions and provide a more comprehensive input basis for subsequent strategy generation.

[0071] In one embodiment, a first hot / cold determination result is generated based on an association strength assessment of the graph structure. More specifically, as... Figure 2 As shown, the method for generating the first hot / cold determination result includes:

[0072] S201: Treat each data item as a node. If any two data items are accessed consecutively by the same access subject within a preset time window, a connection relationship is established between the corresponding nodes. The weight of the connection relationship is the number of times the consecutive access behavior occurs.

[0073] S202: The system maintains an access association graph structure to record the co-occurrence relationships between data items. For each node, it aggregates the access frequency, connection weight, and access sequence information of its neighboring nodes, and calculates the influence of neighboring nodes on the current node through a weighted summation method to obtain node fusion information;

[0074] S203: Combine the original node information with the fused information to generate a popularity score as the first hot / cold determination result. The original node information includes at least one of the following: historical access frequency of the data item, average access duration, last access timestamp, and metadata type identifier.

[0075] The length of the preset time window can be configured according to the business scenario, such as 5 minutes, 15 minutes, or 30 minutes. The establishment of a connection must meet the "continuous access" condition, meaning the time interval between accesses to two data items is less than a set threshold, and the accessing entities are the same.

[0076] This method can identify potential popularity generated by user behavior path dependencies. For example, in an e-commerce platform, if a user frequently visits product A before visiting product B, product B may receive a higher popularity rating due to the associated path.

[0077] In one embodiment, a scheme for generating a second hot / cold determination result based on time series prediction is also included. Details are as follows... Figure 3 As shown, the method for generating the second hot / cold determination result includes:

[0078] S301: Predict future access behavior based on the historical access sequence of data items. The historical access sequence includes multiple access features arranged in chronological order and is extracted to form an input sequence using a sliding window method.

[0079] The historical access sequence includes features such as the number of accesses per unit time, concurrent accesses, and the distribution of access sources. The sliding window has a fixed time interval, such as 5 minutes or 1 hour, with a step size of 1 time unit. After feature normalization, the input sequence is fed into the time series analysis model for prediction. The prediction result is the access probability or trend classification of the data item in the future.

[0080] S302: Convert the prediction results into popularity labels based on a set threshold. For example, mark data items with an access probability higher than 0.7 as "hot", those lower than 0.3 as "cold", and those in between as "warm", as the second hot / cold determination result. Alternatively, convert the prediction results into popularity labels as the second hot / cold determination result. The popularity labels are used to guide the admission strategy of adaptive replacement cache ARC, where data items predicted as hot skip the L1 cache admission evaluation of ARC and are directly written to the head of the T2 queue.

[0081] The step size and length of the sliding window are configurable to adapt to different business cycles. For example, a 5-minute window can be used in a financial transaction system, while a 1-hour window can be used in a content recommendation system.

[0082] In one embodiment, a business rule fusion mechanism based on priority ranking is also included. Specifically, as follows... Figure 4 As shown,

[0083] S401: When generating the final storage policy, the preset business rules are prioritized. Rule categories include compliance rules, security rules, and performance rules, each assigned a highest, second-highest, and lowest priority identifier, respectively. Compliance rules are assigned the highest priority and are used to force specific data to be stored on a specified medium; security rules are assigned the second-highest priority; and performance rules are assigned the lowest priority.

[0084] S402: During policy generation, rules are matched sequentially according to priority. Once a rule is successfully matched, its corresponding policy action is applied directly, and subsequent matching of lower-priority rules is not executed. Compliance rules include mandating the storage of financial audit data on low-speed storage media, or mandating the caching of high-value unpaid orders on high-speed storage media.

[0085] Compliance rules include: financial audit-related data must be stored on low-speed storage media with high durability; security rules include: high-value unpaid orders must be cached on high-speed storage media; performance rules include: data accessed less frequently than a set threshold can be archived to cold storage.

[0086] In other words, the method for generating the final storage policy by combining preset business rules includes: prioritizing the preset business rules, assigning the highest priority identifier to compliance rules, followed by security rules, and the lowest priority to performance rules.

[0087] The business rules stored in the rule base include: financial audit data must be stored on low-speed but high-durability storage media; high-value unpaid orders must be cached on high-speed storage media; data accessed less than a set threshold can be archived to cold storage. When generating policies, the rule decision module matches rules sequentially according to their priority identifiers. Once a rule is successfully matched, its corresponding policy action is applied, and subsequent matching of lower-priority rules is no longer executed.

[0088] This rule matching mechanism ensures that high-priority business constraints take effect first during the policy generation process.

[0089] In one embodiment, a mechanism for manual intervention and audit evidence preservation under compliance trigger conditions is also included. Specifically, when preset compliance trigger conditions are met, a manual intervention process is initiated. These preset compliance trigger conditions include data items carrying financial audit, privacy protection, or anti-money laundering labels.

[0090] When the rule decision module detects that a data item carries the above-mentioned tags and the access behavior meets the definition of an abnormal pattern, a manual intervention process is initiated. Abnormal patterns include: the number of accesses within a unit of time exceeds the threshold, the access source IP is located in an unauthorized area, and the access behavior occurs outside of working hours.

[0091] Once the manual intervention process is initiated, the system pauses automated scheduling and sends a notification to the designated reviewer. During the review process, the system generates an audit log, recording the reviewer's identity, the start and end times of the review, the content of the operation, and the basis for the decision. After the audit log is generated, key fields are extracted to calculate the SHA-256 hash value, which is then submitted to the blockchain node for evidence storage via the API interface. After the blockchain node returns transaction confirmation information, the system records the successful evidence storage status.

[0092] After the blockchain node returns transaction confirmation information, the system records the successful notarization status. This log is used for subsequent auditing and traceability.

[0093] In one embodiment, a method for managing hot and cold data is proposed, comprising:

[0094] Acquire data access events and generate behavioral information to characterize data access behavior based on the data access events. The behavioral information includes multiple of the following: access subject identifier, accessed data item identifier, access timestamp, access type, and access duration.

[0095] Based on the behavioral information, a first hot / cold determination result and a second hot / cold determination result are generated by the first prediction channel and the second prediction channel executed in parallel. The first hot / cold determination result is a popularity score generated based on the access relationship between data items, and the second hot / cold determination result is a popularity tag generated based on the prediction of historical access sequence.

[0096] The first and second cold / hot determination results are fused together to make a decision, and the final storage strategy is generated by combining them with preset business rules. The preset business rules are sorted in advance according to the priority of compliance, security and performance categories, and the rules are matched in order of priority from high to low when the strategy is generated.

[0097] Data storage scheduling operations are performed according to the final storage policy.

[0098] Specifically, generating the first hot / cold determination result includes:

[0099] Each data item is treated as a node. If any two data items are accessed consecutively by the same access subject within a preset time window, a connection relationship is established between the corresponding nodes. The weight of the connection relationship is the number of consecutive accesses.

[0100] Aggregate the associated node information of each node, determine the degree of influence of each associated node on the current node through a preset association strength evaluation method, and obtain the node fusion information by weighted summation;

[0101] The original node information is combined with the fused information to generate a popularity score as the first hot / cold determination result. The original node information includes the historical access frequency, average access duration, last access timestamp, and metadata type identifier of the data item.

[0102] Specifically, the generation of the second hot / cold determination result includes:

[0103] Predict future access behavior based on the historical access sequence of data items. The historical access sequence includes multiple access features arranged in chronological order and is truncated to form a fixed-length input sequence using a sliding window method. The length and step size of the sliding window are configured according to the business scenario.

[0104] The prediction results are converted into heat labels based on a preset threshold as the second hot / cold determination result. The heat labels are used to guide the admission strategy of adaptive replacement cache ARC. For data items predicted as hot, the L1 cache admission evaluation process of the ARC algorithm is skipped and the data is directly written to the head position of its T2 queue.

[0105] Specifically, the generation of the final storage strategy by combining preset business rules includes:

[0106] Prioritize and sort the preset business rules, assign the highest priority flag to compliance rules, the medium priority flag to security rules, and the lowest priority flag to performance rules;

[0107] When generating a strategy, rules are matched sequentially according to their priority identifiers. When a rule is matched, its corresponding strategy action is adopted and the matching of subsequent low-priority rules is terminated. Among them, the compliance rules include: financial audit data is forcibly stored on low-speed storage media; the security rules include: high-value unpaid orders are forcibly cached on high-speed storage media.

[0108] In detail, the methods for managing hot and cold data also include:

[0109] When the preset compliance trigger conditions are met, the manual intervention process is initiated. The preset compliance trigger conditions are that the data item carries a financial audit, privacy protection or anti-money laundering label, and its access behavior conforms to the preset abnormal pattern. The abnormal pattern includes: the number of accesses per unit time exceeds the threshold, the access source IP is located in an unauthorized area, or the access behavior occurs outside of working hours.

[0110] After the manual intervention process is initiated, an audit log is generated containing the identity of the auditor, the audit duration, and the decision basis. The key fields of the audit log are hashed, and the resulting hash value is submitted to the blockchain network for immutable evidence storage.

[0111] In detail, the hot and cold data management method also includes intermediate data secure destruction steps:

[0112] The secure destruction process for intermediate processing data will be initiated when any of the following conditions are met: the predictive processing is completed and the reference count check confirms that the intermediate data has not been referenced by subsequent tasks, the system detects abnormal access behavior, or the manual intervention process is initiated.

[0113] After the destruction is completed, a data destruction certificate is generated. The certificate includes the identifier of the destroyed data, the destruction time, the destruction method, the witness, and the certificate hash value. The certificate hash value is then submitted to the blockchain for evidence storage.

[0114] In one embodiment, such as Figure 5 As shown, a hot and cold data management system is provided, including:

[0115] Feature generation module: used to acquire data access events and generate behavioral information to characterize data access behavior based on the data access events;

[0116] Parallel prediction module: used to generate a first cold / hot determination result and a second cold / hot determination result based on the behavioral information;

[0117] Rule decision module: Used to receive the first cold / hot determination result and the second cold / hot determination result, perform a fusion decision on the two, and generate the final storage strategy in combination with preset business rules;

[0118] Storage scheduling module: used to perform data storage scheduling operations according to the final storage strategy.

[0119] In one embodiment, Figure 5 The system also includes a secure destruction module (not shown in the figure) for securely destroying intermediate processing data. This intermediate processing data includes feature vectors, intermediate prediction results, temporary nodes in the graph structure, and other temporary data generated during system operation. The secure destruction module initiates the destruction process when any of the following conditions are met: prediction processing is complete and the intermediate data is not referenced by subsequent tasks; the system detects abnormal access behavior; or a manual intervention process is initiated.

[0120] Furthermore, after the destruction operation is completed, the system generates a data destruction certificate. This certificate includes the name of the destroyed field, the destruction time, the destruction method (e.g., multiple overwrites, physical destruction), the witness identifier, and the certificate hash. The certificate hash is submitted to a consortium blockchain or public blockchain node for notarization via a blockchain interface, ensuring the immutability and traceability of the destruction operation.

[0121] In one embodiment, the blockchain evidence storage mechanism is implemented by calling a unified blockchain gateway API, which supports and adapts to multiple mainstream blockchain platforms, such as FISCO BCOS and Hyperledger Fabric, thereby ensuring the system's compatibility and scalability.

[0122] In one embodiment, the "abnormal mode" in the manual intervention process includes, but is not limited to: the number of accesses per unit time exceeding a dynamic threshold (such as 3 times the standard deviation based on a historical baseline), the access source IP being located in a high-risk region, or the access behavior occurring during a pre-set non-working period. The system supports configuring specific rules and thresholds for the abnormal mode through a management interface to adapt to the security management needs of different business scenarios.

[0123] Generate a data destruction certificate, which includes the fields to be destroyed, witnesses, destruction method and certificate hash, and submit the certificate hash to the blockchain for evidence storage;

[0124] The secure destruction module initiates the destruction process when any of the following conditions are met: the predictive processing is completed and the intermediate data is not referenced by subsequent tasks, the system detects abnormal access behavior, or the manual intervention process is initiated.

[0125] An intermediate data security destruction mechanism is introduced. When the data processing task is completed or a security rule is triggered, the intermediate data in memory is completely destroyed and an auditable destruction certificate is generated. Blockchain technology is used to ensure the authenticity and non-repudiation of the destruction behavior, effectively solving the risk of information residue.

[0126] In addition, a secure destruction module is used to securely destroy intermediate processed data; generate a data destruction certificate, which includes the destroyed field, witness, destruction method and certificate hash, and submit the certificate hash to the blockchain for notarization; the system improves the accuracy of hot and cold data determination and the rationality of strategy generation through dual prediction channel fusion and rule priority scheduling, and at the same time, through blockchain notarization and secure destruction mechanism, it builds a complete data security and compliance protection system.

[0127] In one embodiment, the rule decision module is further configured to:

[0128] When the first cold / hot determination result is inconsistent with the second cold / hot determination result, the conflict arbitration mechanism is activated;

[0129] The conflict arbitration mechanism includes: obtaining the business attribute tag of the data item; if the business attribute tag belongs to a preset high priority category, then the first cold / hot determination result is adopted; otherwise, the second cold / hot determination result is adopted.

[0130] The modules interact with each other via message queues or remote call interfaces. The system supports deployment on a single server or in a multi-node cluster environment.

[0131] In one embodiment, a dual-channel structure design for the parallel prediction module is also included. The parallel prediction module includes a first prediction channel and a second prediction channel. The first prediction channel generates a popularity score based on the access correlation between data items; the second prediction channel predicts future access behavior based on historical access sequences.

[0132] The two prediction channels operate independently and are not dependent on each other. There is no direct data dependency between the channels, supporting parallel computation. The first prediction channel outputs a popularity score, and the second prediction channel outputs a popularity tag. Both outputs are submitted to the rule decision module through a unified interface.

[0133] In one embodiment, a detailed scheme for generating a popularity score for the first prediction channel is also included. Specifically, the first prediction channel generates a popularity score in the following manner: Each data item is treated as a node; if any two data items are accessed consecutively by the same user within a preset time window, a connection is established between the corresponding nodes, with the weight of the connection being the number of consecutive accesses; the associated node information of each node is aggregated, and the influence of each associated node on the current node is determined using a preset association strength evaluation method; node fusion information is obtained through weighted summation; the original node information and the fusion information are combined to generate a popularity score.

[0134] The original node information includes the historical access frequency, average access duration, last access timestamp, and metadata type identifier for each data item. The metadata type identifier is used to distinguish data categories, such as user information, transaction records, and log files.

[0135] In one embodiment, a historical access sequence modeling method for a second prediction channel is also included. Specifically, the second prediction channel is used to predict future access behavior of data items based on their historical access sequences; the historical access sequences include multiple access features arranged in chronological order and are truncated to form an input sequence using a sliding window method.

[0136] The input sequence has a fixed length, for example, containing access data from the most recent 24 time windows. The system configures the window length and sliding step size according to the business type. After feature normalization, the input sequence is fed into the time series model for prediction.

[0137] The system configures the window length and sliding step size according to the business type. After feature normalization, the input sequence is fed into the time series model for prediction.

[0138] In one embodiment, the application of heat labels in the Adaptive Replacement Cache (ARC) is also included. Specifically, the heat labels output by the second prediction channel are used to guide the admission strategy of the Adaptive Replacement Cache (ARC). For data items predicted as "hot," the admission evaluation process of the L1 cache is skipped when writing to the cache, and the data is directly inserted at the head of the T2 queue.

[0139] The T2 queue is used to store frequently accessed data over a long period of time. This operation ensures that data with high predictability will reside in the long-term cache first.

[0140] In one embodiment, specific examples and priority execution of business rules are also included. Specifically, preset business rules include: financial audit data being forcibly stored on low-speed storage media, or high-value unpaid orders being forcibly cached on high-speed storage media.

[0141] When loading rules, the rule decision module assigns a priority identifier to each rule. During policy generation, the rule base is traversed in priority order, and the rule that matches successfully for the first time is executed.

[0142] In one embodiment, a triggering mechanism for the manual intervention process is also included. Specifically, the rule decision module is further configured to initiate the manual intervention process when preset compliance triggering conditions are met; the preset compliance triggering conditions include data items carrying financial audit, privacy protection, or anti-money laundering labels.

[0143] After receiving the metadata tag of a data item, the rule decision module determines whether it belongs to a preset sensitive category. If so, it further checks whether the access context meets the abnormal mode definition. If it does, it initiates the manual intervention process.

[0144] In one embodiment, the method further includes an audit log for the manual intervention process and a blockchain-based evidence storage method. Specifically, the manual intervention process includes generating an audit log containing the auditor's identity, audit duration, and decision-making basis; and submitting the key hash value of the audit log to the blockchain for immutable evidence storage.

[0145] The audit log includes the auditor's employee ID, the time range of the operation, the processing comments, and a snapshot of the system context. The hash value is generated by calling the hash calculation module and submitted to the consortium blockchain node through the blockchain gateway.

[0146] After the log is generated, the hash calculation module is invoked to generate a SHA-256 hash value, which is then submitted to the blockchain gateway via the API interface. Once the blockchain gateway returns the transaction hash, the system records it in the operation log.

[0147] In one embodiment, a secure destruction mechanism for intermediate data is also included. Specifically, the system further includes a secure destruction module for securely destroying intermediate processed data; generating a data destruction certificate, the certificate including the destroyed field, witness, destruction method, and certificate hash, and storing the certificate hash on the blockchain for evidence.

[0148] Intermediate processing data includes feature vectors, intermediate prediction results, and temporary nodes in the graph structure. After the destruction operation is completed, the system generates a destruction certificate and submits the certificate hash by calling the blockchain interface.

[0149] After the destruction operation is completed, a data destruction certificate is generated, which includes the name of the destroyed field, the destruction time, the destruction method, the witness identifier, and the certificate hash. The certificate hash is stored on the blockchain via the blockchain interface.

[0150] In one embodiment, the method for constructing node information for the first prediction channel is also included. Specifically, when generating a popularity score, the node information used by the first prediction channel includes: the historical access frequency of the data item, the average access duration, the last access timestamp, and the metadata type identifier.

[0151] When generating popularity scores, the first prediction channel relies on node information including: the historical access frequency of data items, average access duration, last access timestamp, and metadata type identifier.

[0152] The above information serves as the initial feature vector for the node and participates in subsequent fusion calculations.

[0153] In one embodiment, the triggering conditions for the secure destruction module are as follows: The secure destruction module initiates the destruction process when any of the following conditions are met: predictive processing is completed and intermediate data is not referenced by subsequent tasks; the system detects abnormal access behavior; or a manual intervention process is initiated.

[0154] The system uses a reference counting mechanism to determine whether intermediate data is referenced, and triggers destruction when the reference count is zero.

[0155] In one embodiment, an arbitration mechanism is also included to handle conflicts in the hot / cold data determination results. Specifically, when the first hot / cold data determination result and the second hot / cold data determination result are inconsistent, the rule decision module initiates the conflict arbitration mechanism, which includes: obtaining the business attribute tag of the data item; if the business attribute tag belongs to a preset high-priority category, such as "core transaction" or "user master data", then the first hot / cold data determination result is adopted; otherwise, the second hot / cold data determination result is adopted.

[0156] In one embodiment, a server 60 is provided, such as Figure 6 As shown, it includes a memory 62, a processor 61, and a computer program stored in the memory 62 and executable on the processor 61. When the processor 61 executes the program, it implements the hot and cold data management method described in any of the foregoing embodiments.

[0157] Server 60 communicates with other system components through a network interface, supporting high-concurrency data processing tasks.

[0158] This invention achieves dynamic management of data storage status through multi-dimensional behavior analysis, dual-channel judgment, rule priority matching, and manual intervention mechanisms, and is applicable to business scenarios with complex data access characteristics such as finance, e-commerce, and cloud computing.

[0159] It should be understood that the technical features in the above embodiments can be combined or substituted with each other to form different implementation methods. For example, the above graph-based heat scoring method can be combined with the above time series prediction method to form a dual-channel judgment mechanism; the above rule priority ranking mechanism can be used in conjunction with the above manual intervention process to achieve dynamic response to compliance rules; the above secure destruction module can be integrated with the above system architecture to manage intermediate data generated during parallel prediction and rule decision-making.

[0160] The combination of features between different embodiments does not depart from the technical concept of this invention and all fall within the protection scope of this invention. By flexibly configuring the functions and parameters of each module, this invention can adapt to the cold and hot data management needs in different business scenarios.

[0161] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for managing hot and cold data, characterized in that, include: Acquire data access events and generate behavioral information to characterize data access behavior based on the data access events; A first cold / hot determination result and a second cold / hot determination result are generated based on the behavioral information; The first and second cold / hot determination results are fused together to make a decision, and the final storage strategy is generated in combination with preset business rules. Data storage scheduling operations are performed according to the final storage policy.

2. The method as described in claim 1, characterized in that, The generation of the first hot / cold determination result includes: Each data item is treated as a node. If any two data items are accessed consecutively by the same object within a preset time window, a connection is established between the corresponding nodes. The weight of the connection is the number of consecutive accesses. Aggregate the associated node information of each node, determine the degree of influence of each associated node on the current node through a preset association strength evaluation method, and obtain the node fusion information by weighted summation; The original node information is combined with the fused information to generate a popularity score, which serves as the first hot / cold determination result.

3. The method as described in claim 1, characterized in that, The generation of the second hot / cold determination result includes: Predict future access behavior based on the historical access sequence of data items. The historical access sequence includes multiple access features arranged in chronological order and is truncated to form an input sequence using a sliding window method. The prediction results are converted into heat labels as the second hot / cold determination result.

4. The method as described in claim 1, characterized in that, The process of generating the final storage strategy by combining preset business rules includes: Prioritize the execution of preset business rules, including assigning the highest priority to compliance rules, followed by security rules, and the lowest priority to performance rules; When generating the strategy, rules are matched according to priority identifiers. Compliance rules include mandating that financial audit data be stored on low-speed storage media, or that high-value unpaid orders be cached on high-speed storage media.

5. The method as described in claim 1, characterized in that, When preset compliance trigger conditions are met, a manual intervention process is initiated. The preset compliance trigger conditions include data items carrying financial audit, privacy protection, or anti-money laundering labels. Once the manual intervention process is initiated, an audit log is generated containing the identity of the auditor, the audit duration, and the basis for the decision. The key hash value of the audit log is then submitted to the blockchain for immutable evidence storage.

6. A cold and hot data management system, characterized in that, include: The feature generation module is used to acquire data access events and generate behavioral information to characterize data access behavior based on the data access events; The parallel prediction module is used to generate a first cold / hot determination result and a second cold / hot determination result based on the behavioral information; The rule decision module is used to receive the first cold / hot determination result and the second cold / hot determination result, perform a fusion decision on the two, and generate the final storage strategy in combination with preset business rules; The storage scheduling module is used to perform data storage scheduling operations according to the final storage strategy.

7. The system as described in claim 6, characterized in that, The parallel prediction module includes a first prediction channel and a second prediction channel. The first prediction channel is used to generate a popularity score based on the access correlation between data items. The popularity score is a numerical value that represents the likelihood of a data item being accessed in the future.

8. The system as described in claim 7, characterized in that, The first prediction channel generates a popularity score in the following way: Each data item is treated as a node. If any two data items are accessed consecutively by the same object within a preset time window, a connection is established between the corresponding nodes. The weight of the connection is the number of consecutive accesses. Aggregate the associated node information of each node, determine the degree of influence of each associated node on the current node through a preset association strength evaluation method, and obtain the node fusion information by weighted summation; The original node information is combined with the fused information to generate a popularity score.

9. The system as described in claim 6, characterized in that, The second prediction channel is used to predict future access behavior based on the historical access sequence of data items; The historical access sequence includes multiple access features arranged in chronological order, and is extracted to form the input sequence using a sliding window method.

10. The system as described in claim 9, characterized in that, The access prediction results output by the second prediction channel are converted into heat labels as the second hot / cold determination results, which are used to guide the admission strategy of Adaptive Replacement Cache (ARC). Data items predicted as "hot" skip the L1 cache admission evaluation process of ARC and are directly written to the head position of the T2 queue.

11. The system as described in claim 6, characterized in that, The preset business rules include mandating that financial audit data be stored on low-speed storage media, or that high-value unpaid orders be cached on high-speed storage media; The rule decision module prioritizes preset business rules, including assigning the highest priority identifier to compliance rules, followed by security rules, and the lowest priority to performance rules, and matching rules according to the priority identifier order when generating the policy.

12. The system as described in claim 6, characterized in that, The rule decision module is also used to initiate a manual intervention process when preset compliance trigger conditions are met; The preset compliance trigger conditions include data items carrying labels related to financial auditing, privacy protection, or anti-money laundering.

13. The system as described in claim 12, characterized in that, The manual intervention process includes generating audit logs containing the identity of the auditors, audit duration, and decision-making basis; Specifically, the key hash value of the audit log is submitted to the blockchain for immutable evidence storage.

14. The system as described in claim 12, characterized in that, The system also includes a secure destruction module for securely destroying intermediate processed data; This involves generating a data destruction certificate, which includes the fields to be destroyed, witnesses, destruction methods, and certificate hashes, and storing the certificate hashes on the blockchain.

15. The system as described in claim 7, characterized in that, The node information used by the first prediction channel in generating the popularity score includes: Historical access frequency of data items; Average access time for data items; The last access timestamp of the data item; Metadata type identifier for data items.

16. The system as described in claim 14, characterized in that, The secure destruction module initiates the destruction process when any of the following conditions are met: Prediction processing is complete and intermediate data is not referenced by subsequent tasks; The system detected abnormal access behavior; The manual intervention process has been initiated.

17. The system as described in claim 6, characterized in that, When the first and second cold / hot determination results are inconsistent, the rule decision module initiates a conflict arbitration mechanism, including: Retrieve the business attribute tags of data items; If the business attribute tag belongs to a preset high priority category, then the first hot / cold determination result is adopted; Otherwise, the second cold / hot determination result shall be adopted.

18. A server, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 5.

Images