Big model based cold and hot data storage optimization method

Abstract

The present application relates to the field of hot and cold data storage, and particularly to a hot and cold data storage optimization method based on a large model, comprising a business integration module, a data dependency module, a mode updating module, a hot and cold partition module and a file merging module, the business integration module is used for collecting business data to form a training data set, the data dependency module is used for fitting business data to obtain a data blood relationship model, the mode updating module is used for predicting an access mode, the hot and cold partition module is used for calculating the heat of data and performing dynamic scheduling, and the file merging module is used for automatically merging small files, the present application can reduce repeated storage, ensure data source uniqueness, standardize data storage structure, reduce storage space, can realize more efficient and more accurate hot and cold data classification, improve resource utilization of data storage hardware, reduce hot data response delay, significantly improve system storage utilization, and improve storage efficiency.

Description

Technical Field

[0001] This invention relates to the field of cold and hot data storage, specifically to a method for optimizing cold and hot data storage based on a large model. Background Technology

[0002] Hot and cold data storage is a data management method that employs different storage strategies based on the frequency and importance of data access. By dividing data into frequently accessed "hot" data and infrequently accessed "cold" data, the data access characteristics are matched with the hardware storage performance, maximizing data storage efficiency. Currently, the storage-compute separation architecture based on hot and cold data has become the mainstream solution in cloud computing and big data processing.

[0003] Traditional cold and hot data storage solutions classify data based on access frequency or time windows. In complex computer systems, this approach struggles to accurately adapt to different business scenarios, and the data dependencies between classification tasks are difficult to track, leading to wasted storage resources and difficulties in dynamically adjusting optimization strategies. When business data changes frequently, the cold and hot data switching cycle shortens accordingly, and data dependencies also change constantly. Traditional cold and hot data classification algorithms cannot adapt to high-concurrency business data environments.

[0004] Furthermore, in business processing scenarios such as log storage, stream computing, and transaction data, a large number of small files such as log data and transaction data are often generated. These small files not only increase the difficulty of storage management, but also lead to frequent disk read and write operations, increasing storage management and query overhead. However, simple compression algorithms cannot be effectively adjusted according to the actual access patterns, resulting in the storage resources not being used optimally and affecting the normal operation of the database. Summary of the Invention

[0005] The purpose of this invention is to provide a method for optimizing cold and hot data storage based on a large model, so as to solve the problems mentioned in the background art.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a large-model-based cold and hot data storage optimization system, comprising: a business integration module, a data dependency module, a schema update module, a cold and hot partitioning module, and a file merging module;

[0007] The business integration module is used to integrate with various business management systems through API plugins. The business management system includes log analysis, AI training and data lake system, collects business data, including: access logs, storage mode and computing task execution records, defines a unified log format, tracks the data generation, storage, computing and consumption process, and classifies and stores the tracked data in the database to form a training dataset.

[0008] The data dependency module is used to parse the business semantics in the SQL database using a large language model or multimodal model with a GNN+Transformer hybrid architecture, train the model using training datasets from historical periods, fit the relationship between various business data in the business system, the data call chain and upstream and downstream dependency information, perform cross-level classification according to the data call or dependency relationship, and perform same-level classification according to the data relationship to obtain the data lineage model.

[0009] The pattern update module is used to embed and encode data access behavior using the trained large model, capture the pattern features of the data, including: data time series features, periodic features and burst features, predict the user's access pattern in the next period, including: number of accesses, access frequency and operation type, and after the end of the next period, input the collected data changes into the large model to automatically update the hot and cold tags and data storage strategy.

[0010] The hot and cold partitioning module uses a Bayesian algorithm to calculate the popularity of each group of data based on the current time, data creation time, user access patterns, and data lineage model. The data is divided into hot data, warm data, and cold data, and each data unit is labeled with a hot, cold, or warm tag. The data unit includes data tables, data blocks, files, and object storage units. At the same time, a data migration mechanism is initiated. Hot data is stored in high-performance storage media, cold data is stored in low-cost storage media, and warm data is dynamically scheduled according to the predicted user access patterns.

[0011] The file merging module is used to periodically detect small files generated in the business system. These small files include log data and transaction data. The module analyzes their access patterns using a large model, automatically merges the small files according to the data lineage model, and retains the original file's virtual reference layer after merging, so that the business code that has not been updated can still access them.

[0012] Furthermore, the business integration module includes: a data collection unit and a flow tracking unit;

[0013] The data collection unit is used to collect and adapt data through Kafka, CDC or ETL tools, acquire business data and clean, standardize, time-align and structure the raw data;

[0014] The flow tracking unit is used to parse SQL scripts using a large model, track implicit dependencies in database JOIN or WHERE operations, and build a training dataset.

[0015] Furthermore, the data dependency module includes: a large model unit, a training data unit, and a lineage model unit;

[0016] The large model unit is used to build or call a pre-trained large model and use the large model to analyze the collected data.

[0017] The training data unit is used to pre-train the large model by predicting data entity masks or relational paths, and to perform secondary training on the large model using the training dataset.

[0018] The lineage model unit is used to establish the dependency relationship between data and business modules and the access heat propagation path through embedded analysis of business data, and generate a data lineage model.

[0019] Furthermore, the pattern update module includes: a feature capture unit and a strategy adjustment unit;

[0020] The feature capture unit is used to capture the pattern features of the data and predict the user's access pattern.

[0021] The strategy adjustment unit is used to periodically recollect business data and input it into the large model for retraining and inference.

[0022] Furthermore, the hot and cold partitioning module includes: a heat calculation unit, a data migration unit, and a dynamic scheduling unit;

[0023] The heat calculation unit is used to output temperature status labels and temperature trend predictions for each data point from the large model.

[0024] The data migration unit is used to tag data with cold, hot, and temperature labels based on the heat calculation results and allocate them to different storage media;

[0025] The dynamic scheduling unit is used to enable the data migration mechanism, including hierarchical storage, periodic preheating, and delayed loading, to ensure data consistency.

[0026] Furthermore, the file merging module includes: a small file retrieval unit and an association merging unit;

[0027] The small file retrieval unit is used to retrieve small files and output the file size, level in the data lineage model, and number of local accesses of the small files.

[0028] The associated merging unit is used to calculate the percentage decrease in NameNode pressure before and after merging, and to make intelligent decisions on merging small files.

[0029] The method for optimizing cold and hot data storage based on a large model includes the following steps:

[0030] Step S1. Integrate the API plugin into the business management system to track the data generation, storage, calculation and consumption process, collect business data logs, classify and store the business data in the database according to a unified log format, and obtain the training dataset;

[0031] Step S2. Train a large model using the training dataset. The large model parses the business semantics in the SQL database, fits the relationships between various business data, data call chains and upstream and downstream dependency information, sorts data across levels based on data calls or dependencies, and sorts data within the same level based on data relationships to obtain a data lineage model.

[0032] Step S3. Use the trained large model to embed and encode data access behavior, capture the pattern features of business data, predict the user's access pattern in the next cycle, and update the large model after the next cycle ends.

[0033] Step S4. Based on the current time, data creation time, user access pattern, and data lineage model, determine the popularity of each group of data, label it as hot data, warm data, or cold data, start the data migration mechanism, store hot data in high-performance storage media, store cold data in low-cost storage media, and dynamically schedule warm data.

[0034] Step S5. Periodically detect small files generated in the business system, analyze their access patterns using a large model, automatically merge excessive small files based on the data lineage model, and retain the virtual reference layer of the original files after merging.

[0035] Furthermore, step S1 includes:

[0036] Step S11. Connect the cloud and various business systems through the API plugin to collect business data. The business management system includes log analysis, AI training and data lake system. The business data includes: access logs, storage mode and computing task execution records. At the same time, install a low-overhead collection agent plugin in the business system to increase the stability of the data collection process.

[0037] Step S12. Use Kafka, CDC, or ETL tools to collect and adapt data, obtain business data, and clean, standardize, time-align, and structure the raw data. Upload the business data to the SQL database, classify and store the business data according to a unified log format, and obtain the training dataset.

[0038] Furthermore, step S2 includes:

[0039] Step S21. Use a large language model or multimodal model with a GNN+Transformer hybrid architecture to parse the business semantics in the SQL database, train the model using the training dataset within the historical period, fit the relationship between various business data in the business system, the data call chain and upstream and downstream dependency information, and obtain the large model after training.

[0040] Step S22. After training, the large model parses the SQL database script, tracks the implicit dependencies in the database JOIN or WHERE operations, performs lineage analysis on the entire lifecycle of the data, classifies the data across levels according to the data call or dependency relationship, classifies the data at the same level according to the data association relationship, builds a hierarchical data relationship graph starting from the basic data, models the strong and weak dependencies between the data and business modules and the access heat propagation path, and obtains the data lineage model.

[0041] Furthermore, step S3 includes:

[0042] Step S31. Use the trained large model to embed and encode data access behavior. By capturing the pattern features of business data, including data time series features, periodic features and burst features, predict the user's access pattern in the next period. The access pattern includes: number of accesses, access frequency and operation type.

[0043] Step S32. After the next cycle ends, collect business data again, input the collected data changes into the large model for retraining and inference, and automatically update the hot and cold labels and data storage strategy.

[0044] Furthermore, step S4 includes:

[0045] Step S41. Determine the popularity of each group of data, satisfying H=c·e^[-a·(t1-t2)]+m·Hc+n·Hf, where H is the popularity of the data, a is a coefficient related to the historical access frequency, t1 and t2 are the current time and the last access time of the data, m and n are the preset cross-level ratio coefficient and same-level ratio coefficient, Hc is the average popularity of data with call or dependency relationship in the data lineage model, and Hf is the average popularity of data with association relationship in the data lineage model. If there is no call, dependency or association data, then Hc and Hf are both 0.

[0046] Step S42. According to the preset heat range, divide the data into hot data, warm data and cold data, and label each data unit with hot, cold and warm tags. Store hot data in high-performance storage media, including SSD and cache, store cold data in low-cost storage media, including cold backup repository and archive repository, and dynamically schedule warm data according to the predicted user access patterns by introducing tiered storage, periodic preheating and delayed loading mechanisms.

[0047] Furthermore, step S5 includes:

[0048] Step S51. Periodically detect small files generated in the business system, and output the file size, level in the data lineage model, and number of local accesses of the small files. The small files include log data and transaction data.

[0049] Step S52. Calculate the percentage decrease in NameNode pressure before and after the merge, make intelligent decisions on merging small files, and retain the original file virtual reference layer after the merge so that the unupdated business code can still be accessed.

[0050] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0051] This invention tracks the generation, storage, computation, and consumption of business data by collecting access logs, storage patterns, and computation task execution records. This allows for the analysis of the lineage of business data, determining the dependencies between data, reducing redundant storage, ensuring the uniqueness of the data source, standardizing the data storage structure, reducing storage space, and enabling more efficient and accurate classification of hot and cold data.

[0052] This invention embeds and encodes data access behavior using a large model, captures data pattern features, predicts user access patterns, calculates the popularity of each data group, and classifies and stores data into hot data, warm data, and cold data. This reduces data storage costs, optimizes database access performance, improves the resource utilization of data storage hardware, and reduces hot data response latency.

[0053] This invention analyzes the access patterns of small files generated in the business system using a large model, automatically merges excessive small files based on data lineage, updates cold and hot tags and data storage strategies, and automatically adjusts storage strategies. It can intelligently optimize data storage layout according to the needs of different business scenarios. Through precise hierarchical storage of cold and hot data, optimization of small file merging, and adaptive storage adjustment strategies, it significantly improves system storage utilization and enhances storage efficiency. Attached Figure Description

[0054] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0055] Figure 1 This is a schematic diagram of the structure of the cold and hot data storage optimization system based on a large model according to the present invention;

[0056] Figure 2 This is a schematic diagram illustrating the steps of the cold and hot data storage optimization method based on a large model according to the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Please see Figure 1 The present invention provides a technical solution: a cold and hot data storage optimization system based on a large model, comprising: a business integration module, a data dependency module, a pattern update module, a cold and hot partitioning module, and a file merging module;

[0059] The business integration module is used to integrate with various business management systems through API plugins. The business management system includes log analysis, AI training and data lake system, collects business data, including: access logs, storage mode and computing task execution records, defines a unified log format, tracks the data generation, storage, computing and consumption process, and classifies and stores the tracked data in the database to form a training dataset.

[0060] The business integration module includes: a data collection unit and a flow tracking unit;

[0061] The data collection unit is used to collect and adapt data through Kafka, CDC or ETL tools, acquire business data and clean, standardize, time-align and structure the raw data;

[0062] The flow tracking unit is used to parse SQL scripts using a large model, track implicit dependencies in database JOIN or WHERE operations, and build a training dataset.

[0063] The data dependency module is used to parse the business semantics in the SQL database using a large language model or multimodal model with a GNN+Transformer hybrid architecture, train the model using training datasets from historical periods, fit the relationship between various business data in the business system, the data call chain and upstream and downstream dependency information, perform cross-level classification according to the data call or dependency relationship, and perform same-level classification according to the data relationship to obtain the data lineage model.

[0064] The data dependency module includes: a large model unit, a training data unit, and a lineage model unit;

[0065] The large model unit is used to build or call a pre-trained large model and use the large model to analyze the collected data.

[0066] The training data unit is used to pre-train the large model by predicting data entity masks or relational paths, and to perform secondary training on the large model using the training dataset.

[0067] The lineage model unit is used to establish the dependency relationship between data and business modules and the access heat propagation path through embedded analysis of business data, and generate a data lineage model.

[0068] The pattern update module is used to embed and encode data access behavior using the trained large model, capture the pattern features of the data, including: data time series features, periodic features and burst features, predict the user's access pattern in the next period, including: number of accesses, access frequency and operation type, and after the end of the next period, input the collected data changes into the large model to automatically update the hot and cold tags and data storage strategy.

[0069] The mode update module includes: a feature capture unit and a strategy adjustment unit;

[0070] The feature capture unit is used to capture the pattern features of the data and predict the user's access pattern.

[0071] The strategy adjustment unit is used to periodically recollect business data and input it into the large model for retraining and inference.

[0072] The hot and cold partitioning module uses a Bayesian algorithm to calculate the popularity of each group of data based on the current time, data creation time, user access patterns, and data lineage model. The data is divided into hot data, warm data, and cold data, and each data unit is labeled with a hot, cold, or warm tag. The data unit includes data tables, data blocks, files, and object storage units. At the same time, a data migration mechanism is initiated. Hot data is stored in high-performance storage media, cold data is stored in low-cost storage media, and warm data is dynamically scheduled according to the predicted user access patterns.

[0073] The hot and cold partitioning module includes: a heat calculation unit, a data migration unit, and a dynamic scheduling unit;

[0074] The heat calculation unit is used to output temperature status labels and temperature trend predictions for each data point from the large model.

[0075] The data migration unit is used to tag data with cold, hot, and temperature labels based on the heat calculation results and allocate them to different storage media;

[0076] The dynamic scheduling unit is used to enable the data migration mechanism, including hierarchical storage, periodic preheating, and delayed loading, to ensure data consistency.

[0077] The file merging module is used to periodically detect small files generated in the business system. These small files include log data and transaction data. The module analyzes their access patterns using a large model, automatically merges the small files according to the data lineage model, and retains the original file's virtual reference layer after merging, so that the business code that has not been updated can still access them.

[0078] The file merging module includes: a small file retrieval unit and an associated merging unit;

[0079] The small file retrieval unit is used to retrieve small files and output the file size, level in the data lineage model, and number of local accesses of the small files.

[0080] The associated merging unit is used to calculate the percentage decrease in NameNode pressure before and after merging, and to make intelligent decisions on merging small files.

[0081] like Figure 2 As shown, the cold and hot data storage optimization method based on a large model includes the following steps:

[0082] Step S1. Integrate the API plugin into the business management system to track the data generation, storage, calculation and consumption process, collect business data logs, classify and store the business data in the database according to a unified log format, and obtain the training dataset;

[0083] Step S1 includes:

[0084] Step S11. Connect the cloud and various business systems through the API plugin to collect business data. The business management system includes log analysis, AI training and data lake system. The business data includes: access logs, storage mode and computing task execution records. At the same time, install a low-overhead collection agent plugin in the business system to increase the stability of the data collection process.

[0085] Step S12. Use Kafka, CDC, or ETL tools to collect and adapt data, obtain business data, and clean, standardize, time-align, and structure the raw data. Upload the business data to the SQL database, classify and store the business data according to a unified log format, and obtain the training dataset.

[0086] Step S2. Train a large model using the training dataset. The large model parses the business semantics in the SQL database, fits the relationships between various business data, data call chains and upstream and downstream dependency information, sorts data across levels based on data calls or dependencies, and sorts data within the same level based on data relationships to obtain a data lineage model.

[0087] Step S2 includes:

[0088] Step S21. Use a large language model or multimodal model with a GNN+Transformer hybrid architecture to parse the business semantics in the SQL database, train the model using the training dataset within the historical period, fit the relationship between various business data in the business system, the data call chain and upstream and downstream dependency information, and obtain the large model after training.

[0089] Step S22. After training, the large model parses the SQL database script, tracks the implicit dependencies in the database JOIN or WHERE operations, performs lineage analysis on the entire lifecycle of the data, classifies the data across levels according to the data call or dependency relationship, classifies the data at the same level according to the data association relationship, builds a hierarchical data relationship graph starting from the basic data, models the strong and weak dependencies between the data and business modules and the access heat propagation path, and obtains the data lineage model.

[0090] Step S3. Use the trained large model to embed and encode data access behavior, capture the pattern features of business data, predict the user's access pattern in the next cycle, and update the large model after the next cycle ends.

[0091] Step S3 includes:

[0092] Step S31. Use the trained large model to embed and encode data access behavior. By capturing the pattern features of business data, including data time series features, periodic features and burst features, predict the user's access pattern in the next period. The access pattern includes: number of accesses, access frequency and operation type.

[0093] Step S32. After the next cycle ends, collect business data again, input the collected data changes into the large model for retraining and inference, and automatically update the hot and cold labels and data storage strategy.

[0094] Step S4. Based on the current time, data creation time, user access pattern, and data lineage model, determine the popularity of each group of data, label it as hot data, warm data, or cold data, start the data migration mechanism, store hot data in high-performance storage media, store cold data in low-cost storage media, and dynamically schedule warm data.

[0095] Step S4 includes:

[0096] Step S41. Determine the popularity of each group of data, satisfying H=c·e^[-a·(t1-t2)]+m·Hc+n·Hf, where H is the popularity of the data, a is a coefficient related to the historical access frequency, t1 and t2 are the current time and the last access time of the data, m and n are the preset cross-level ratio coefficient and same-level ratio coefficient, Hc is the average popularity of data with call or dependency relationship in the data lineage model, and Hf is the average popularity of data with association relationship in the data lineage model. If there is no call, dependency or association data, then Hc and Hf are both 0.

[0097] Step S42. According to the preset heat range, divide the data into hot data, warm data and cold data, and label each data unit with hot, cold and warm tags. Store hot data in high-performance storage media, including SSD and cache, store cold data in low-cost storage media, including cold backup repository and archive repository, and dynamically schedule warm data according to the predicted user access patterns by introducing tiered storage, periodic preheating and delayed loading mechanisms.

[0098] Step S5. Periodically detect small files generated in the business system, analyze their access patterns using a large model, automatically merge excessive small files based on the data lineage model, and retain the virtual reference layer of the original files after merging.

[0099] Step S5 includes:

[0100] Step S51. Periodically detect small files generated in the business system, and output the file size, level in the data lineage model, and number of local accesses of the small files. The small files include log data and transaction data.

[0101] Step S52. Calculate the percentage decrease in NameNode pressure before and after the merge, make intelligent decisions on merging small files, and retain the original file virtual reference layer after the merge so that the unupdated business code can still be accessed.

[0102] Example: There are 4 data units in the business system. Using a large model to track access to the data units, it is found that data unit 1 depends on data unit 2, data unit 2 is associated with data unit 3, and data unit 4 has no dependency or association relationship. Therefore, a three-layer data lineage model is constructed, with data unit 1 as the top layer, data units 2 and 3 as the lower layers, and data unit 4 as the bottom layer. The hotness of the 4 data blocks are 0.8, 0.5, 0.7 and 0.2 respectively. Therefore, data units 1 and 3 are stored as hot data in the cache, data unit 2 is scheduled as warm data, and data unit 4 is stored as cold data on the hard disk.

[0103] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0104] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for optimizing cold and hot data storage based on a large model, characterized in that, The method includes the following steps: Step S1. Integrate the API plugin into the business management system to track the data generation, storage, calculation and consumption process, collect business data logs, classify and store the business data in the database according to a unified log format, and obtain the training dataset; Step S2. Train a large model using the training dataset. The large model parses the business semantics in the SQL database, fits the relationships between various business data, data call chains and upstream and downstream dependency information, sorts data across levels based on data calls or dependencies, and sorts data within the same level based on data relationships to obtain a data lineage model. Step S3. Use the trained large model to embed and encode data access behavior, capture the pattern features of business data, predict the user's access pattern in the next cycle, and update the large model after the next cycle ends. Step S4. Based on the current time, data creation time, user access pattern, and data lineage model, determine the popularity of each group of data, label it as hot data, warm data, or cold data, start the data migration mechanism, store hot data in high-performance storage media, store cold data in low-cost storage media, and dynamically schedule warm data. Step S5. Periodically detect small files generated in the business system, analyze their access patterns using a large model, automatically merge excessive small files according to the data lineage model, and retain the virtual reference layer of the original files after merging; Step S4 includes: Step S41. Determine the popularity of each group of data, satisfying H=c·e^[-a·(t1-t2)]+m·Hc+n·Hf, where H is the popularity of the data, a is a coefficient related to the historical access frequency, t1 and t2 are the current time and the last access time of the data, m and n are the preset cross-level ratio coefficient and same-level ratio coefficient, Hc is the average popularity of data with call or dependency relationship in the data lineage model, and Hf is the average popularity of data with association relationship in the data lineage model; Step S42. According to the preset heat range, divide the data into hot data, warm data and cold data, and label each data unit with hot, cold and warm tags. Store hot data in high-performance storage media, including SSD and cache, store cold data in low-cost storage media, including cold backup repository and archive repository, and dynamically schedule warm data according to the predicted user access patterns by introducing tiered storage, periodic preheating and delayed loading mechanisms.

2. The method for optimizing cold and hot data storage based on a large model according to claim 1, characterized in that: Step S1 includes: Step S11. Connect the cloud and various business systems through the API plugin to collect business data. The business management system includes log analysis, AI training and data lake system. The business data includes: access logs, storage mode and computing task execution records. At the same time, install a low-overhead collection agent plugin in the business system to increase the stability of the data collection process. Step S12. Use Kafka, CDC, or ETL tools to collect and adapt data, obtain business data, and clean, standardize, time-align, and structure the raw data. Upload the business data to the SQL database, classify and store the business data according to a unified log format, and obtain the training dataset.

3. The method for optimizing cold and hot data storage based on a large model according to claim 2, characterized in that: Step S2 includes: Step S21. Use a large language model or multimodal model with a GNN+Transformer hybrid architecture to parse the business semantics in the SQL database, train the model using the training dataset within the historical period, fit the relationship between various business data in the business system, the data call chain and upstream and downstream dependency information, and obtain the large model after training. Step S22. After training, the large model parses the SQL database script, tracks the implicit dependencies in the database JOIN or WHERE operations, performs lineage analysis on the entire lifecycle of the data, classifies the data across levels according to the data call or dependency relationship, classifies the data at the same level according to the data association relationship, builds a hierarchical data relationship graph starting from the basic data, models the strong and weak dependencies between the data and business modules and the access heat propagation path, and obtains the data lineage model.

4. The method for optimizing cold and hot data storage based on a large model according to claim 3, characterized in that: Step S3 includes: Step S31. Use the trained large model to embed and encode data access behavior. By capturing the pattern features of business data, including data time series features, periodic features and burst features, predict the user's access pattern in the next period. The access pattern includes: number of accesses, access frequency and operation type. Step S32. After the next cycle ends, collect business data again, input the collected data changes into the large model for retraining and inference, and automatically update the hot and cold labels and data storage strategy.

5. The method for optimizing cold and hot data storage based on a large model according to claim 4, characterized in that: Step S5 includes: Step S51. Periodically detect small files generated in the business system, and output the file size, level in the data lineage model, and number of local accesses of the small files. The small files include log data and transaction data. Step S52. Calculate the percentage decrease in NameNode pressure before and after the merge, make intelligent decisions on merging small files, and retain the original file virtual reference layer after the merge so that the unupdated business code can still be accessed.

6. A large-model-based cold and hot data storage optimization system, wherein the system executes the large-model-based cold and hot data storage optimization method as described in claim 1, characterized in that, It includes the following modules: business integration module, data dependency module, pattern update module, hot and cold partitioning module, and file merging module; The business integration module is used to integrate with various business management systems through API plugins. The business management system includes log analysis, AI training and data lake system, collects business data, including: access logs, storage mode and computing task execution records, defines a unified log format, tracks the data generation, storage, computing and consumption process, and classifies and stores the tracked data in the database to form a training dataset. The data dependency module is used to parse the business semantics in the SQL database using a large language model or multimodal model with a GNN+Transformer hybrid architecture, train the model using training datasets from historical periods, fit the relationship between various business data in the business system, the data call chain and upstream and downstream dependency information, perform cross-level classification according to the data call or dependency relationship, and perform same-level classification according to the data relationship to obtain the data lineage model. The pattern update module is used to embed and encode data access behavior using the trained large model, capture the pattern features of the data, including: data time series features, periodic features and burst features, predict the user's access pattern in the next period, including: number of accesses, access frequency and operation type, and after the end of the next period, input the collected data changes into the large model to automatically update the hot and cold tags and data storage strategy. The hot and cold partitioning module uses a Bayesian algorithm to calculate the popularity of each group of data based on the current time, data creation time, user access patterns, and data lineage model. The data is divided into hot data, warm data, and cold data, and each data unit is labeled with a hot, cold, or warm tag. The data unit includes data tables, data blocks, files, and object storage units. At the same time, a data migration mechanism is initiated. Hot data is stored in high-performance storage media, cold data is stored in low-cost storage media, and warm data is dynamically scheduled according to the predicted user access patterns. The file merging module is used to periodically detect small files generated in the business system. These small files include log data and transaction data. The module analyzes their access patterns using a large model, automatically merges the small files according to the data lineage model, and retains the original file's virtual reference layer after merging, so that the business code that has not been updated can still access them.

7. The large-model-based cold and hot data storage optimization system according to claim 6, characterized in that: The business integration module includes: a data collection unit and a flow tracking unit; The data collection unit is used to collect and adapt data through Kafka, CDC or ETL tools, acquire business data and clean, standardize, time-align and structure the raw data; The flow tracking unit is used to parse SQL scripts using a large model, track implicit dependencies in database JOIN or WHERE operations, and build a training dataset.

8. The cold and hot data storage optimization system based on a large model according to claim 7, characterized in that: The data dependency module includes: a large model unit, a training data unit, and a lineage model unit; The large model unit is used to build or call a pre-trained large model and use the large model to analyze the collected data. The training data unit is used to pre-train the large model by predicting data entity masks or relational paths, and to perform secondary training on the large model using the training dataset. The lineage model unit is used to establish the dependency relationship between data and business modules and the access heat propagation path through embedded analysis of business data, and generate a data lineage model.

9. The cold and hot data storage optimization system based on a large model according to claim 8, characterized in that: The mode update module includes: a feature capture unit and a strategy adjustment unit; The feature capture unit is used to capture the pattern features of the data and predict the user's access pattern. The strategy adjustment unit is used to periodically recollect business data and input it into the large model for retraining and inference. The hot and cold partitioning module includes: a heat calculation unit, a data migration unit, and a dynamic scheduling unit; The heat calculation unit is used to output temperature status labels and temperature trend predictions for each data point from the large model. The data migration unit is used to tag data with cold, hot, and temperature labels based on the heat calculation results and allocate them to different storage media; The dynamic scheduling unit is used to enable the data migration mechanism, including hierarchical storage, periodic preheating, and delayed loading, to ensure data consistency.

10. The large-model-based cold and hot data storage optimization system according to claim 9, characterized in that: The file merging module includes: a small file retrieval unit and an associated merging unit; The small file retrieval unit is used to retrieve small files and output the file size, level in the data lineage model, and number of local accesses of the small files. The associated merging unit is used to calculate the percentage decrease in NameNode pressure before and after merging, and to make intelligent decisions on merging small files.

Images