A data storage method and system based on large model

Abstract

The present invention relates to the technical field of large-scale data management, specifically to a data storage method and system based on a large model, comprising the following steps: collecting multiple data in a large model data set, recording the number of occurrences of each data, calculating the entropy value of each data, determining the volatility and diversity of the data, and generating an entropy value calculation result. In the present invention, by sorting and classifying the data according to the entropy value, high entropy value data is located in a high-density storage area, while low entropy value data is stored in a low-density area, optimizing the utilization of storage space, reducing unnecessary storage costs, simulating stratigraphic superposition according to the hierarchical storage of data and sorting by time tags, not only optimizing the storage structure, but also facilitating the rapid positioning and access of new data, data compression and storage parameter adjustment simulate the compaction process of geological layers, effectively managing storage capacity, enhancing the energy efficiency of data centers, and enhancing the responsiveness to complex data environments.

Description

Technical Field

[0001] The present invention relates to the technical field of large-scale data management, and in particular to a data storage method and system based on a large model. Background Art

[0002] Large-scale data management is an important branch of information technology that focuses on developing and maintaining technologies and methods for processing, storing and analyzing huge data sets. This field covers multiple aspects such as data acquisition, storage, management, analysis and visualization. Key challenges in the field of technology include efficient storage of data, fast query processing and management of real-time data streams. Large-scale data management technology is widely used in cloud computing, big data analysis, Internet technology and enterprise data warehouses, and is the infrastructure that supports modern data-intensive applications.

[0003] Among them, the data storage method of large models refers to the technology of using large machine learning models to optimize the data storage process. This method improves the performance of the storage system and the speed of data access through automated data classification, intelligent compression technology, and efficient data retrieval technology. The main uses include reducing storage space requirements, increasing data processing speed, and optimizing the energy efficiency of data centers. These technologies are particularly suitable for processing large-scale, structured or unstructured data sets, such as social media data, scientific research data, and enterprise-level application data.

[0004] Existing large-scale data management technologies face the dual challenges of efficiency and cost when processing large amounts of diverse data. Conventional data storage methods fail to effectively distinguish the importance and frequency of use of data, resulting in high-value data and low-value data being stored in the same way, increasing ineffective storage space usage and costs. Existing technologies fail to achieve dynamic optimization of data locations during data access and retrieval, resulting in data access delays and reduced processing speed and efficiency. This static storage and access method is particularly inadequate when data volume surges, and it is difficult to adapt to rapidly changing data access needs. For example, a data storage structure that is not frequently updated will result in a data processing system that does not respond in a timely manner when demand changes dramatically, affecting the performance and service quality of the entire data center. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a data storage method and system based on a large model.

[0006] In order to achieve the above object, the present invention adopts the following technical solution, a data storage method based on a large model, comprising the following steps:

[0007] S1: Collect multiple data in the large model data set, record the number of occurrences of each data, calculate the entropy value of each data, determine the volatility and diversity of the data, and generate entropy calculation results;

[0008] S2: Based on the entropy value calculation result, the data is sorted by entropy value using a large model, data items with higher entropy values ​​are located in a high-density storage area, and data items with lower entropy values ​​are located in a low-density storage area, and a hierarchical index for data storage is generated;

[0009] S3: Based on the data storage hierarchical index, the data access frequency of the storage area is calculated, the data storage location is adjusted according to the frequency and entropy value, the physical storage location of the data is optimized, and a storage structure optimization plan is generated;

[0010] S4: using the storage structure optimization scheme, the newly collected data are sequentially added to the top layer of the large model data set, sorted according to the time stamp of the new data, and the new data are preferentially stored in the top layer, simulating the stratigraphic superposition process, and generating data hierarchy update results;

[0011] S5: compressing the data of the lower layer in the data hierarchy update result, simulating the compression density change during the geological layer compaction process, adjusting the storage parameters, and generating a formation compaction simulation result;

[0012] S6: According to the formation compaction simulation result and in combination with the data access frequency, the storage level of the data is dynamically adjusted, the data access path is updated to match the change of the access mode, and a dynamic access plan is generated.

[0013] As a further solution of the present invention, the entropy value calculation result includes a list of entropy values ​​of data items, identifiers of data items, corresponding entropy values ​​and rankings of entropy value sizes; the data storage hierarchical index includes a sorting index of data items, storage levels corresponding to data items and storage density information of each level; the storage structure optimization scheme includes a new configuration of storage locations, a comparison of access frequencies before and after configuration and a capacity change of storage areas; the data hierarchy update result includes the access time of newly added data items, the storage level and the order of data in the level; the formation compaction simulation result includes storage capacity before and after data compression, compression rate and access efficiency after compression; the dynamic access scheme includes adjustment records of data levels, access paths after adjustment and comparison of response times before and after path optimization.

[0014] As a further solution of the present invention, multiple data in the large model data set are collected, the number of occurrences of each data is recorded, the entropy value of each data is calculated, the volatility and diversity of the data are determined, and the steps of generating the entropy value calculation result are specifically as follows:

[0015] S101: extract multiple data types from the large model data set, count the occurrence frequency of each data type, store them in a dictionary in a key-value pair format, and obtain a data type frequency table;

[0016] S102: Based on the data type frequency table, calculate the proportion of the total number of occurrences of each data type to the total number of data sets to obtain a data probability distribution table;

[0017] S103: using the data probability distribution table, calculating the entropy value for each data type according to the probability value, using a method of cumulatively adding the negative value of the probability multiplied by the logarithm to obtain an entropy value calculation result.

[0018] As a further solution of the present invention, according to the entropy value calculation result, the data is sorted by entropy value using a large model, and data items with higher entropy values ​​are located in a high-density storage area, and data items with lower entropy values ​​are located in a low-density storage area. The steps of generating a hierarchical index for data storage are specifically as follows:

[0019] S201: Based on the entropy value calculation result, the data items are screened and classified to identify high entropy value and low entropy value data items, a temporary identifier is assigned to each data item, and it is marked as high priority and low priority, and the high entropy value data is prioritized to generate an entropy value grouping table;

[0020] S202: using the entropy grouping table, for high entropy data items, configuring a storage area with read and write capabilities, migrating the data to a high-density storage area, performing the opposite operation on low-entropy data items, allocating them to a low-density storage area with lower costs, updating the storage configuration, and generating a data allocation record;

[0021] S203: Through the data allocation record, synchronize the access rights and speed settings of the storage area, check the storage location of each data item and the entropy value match, configure the data access path, optimize the data access efficiency, and generate a data storage hierarchical index.

[0022] As a further solution of the present invention, based on the data storage hierarchical index, the data access frequency of the storage area is calculated, the data storage location is adjusted according to the frequency and entropy value, the physical storage location of the data is optimized, and the steps of generating a storage structure optimization solution are specifically as follows:

[0023] S301: Based on the data storage hierarchical index, monitoring data access records of multiple storage areas, calculating access frequencies of data items, and generating data access frequency records;

[0024] S302: using the data access frequency record, combined with the entropy value grouping table, comparing the access frequency and the entropy value, identifying data items with high access frequency and low entropy value, data items with low access frequency and high entropy value, adjusting the storage location, and generating a list of data items to be optimized;

[0025] S303: According to the list of data items to be optimized, reallocate data storage locations, move data with high access frequency to high-speed storage areas, and move data with low access frequency to low-speed storage areas, optimize the physical storage structure of data, and generate a storage structure optimization plan.

[0026] As a further solution of the present invention, the storage structure optimization solution is used to sequentially add the newly collected data to the top layer of the large model data set, sort the data according to the time stamp of the new data, and store the new data in the top layer first, simulating the stratigraphic stacking process, and generating the data hierarchy update result in the following steps:

[0027] S401: According to the storage structure optimization scheme, a strategy for adding data to the large model data set is set, the newly collected data is added to the top layer of the data set, the time stamp of the new data is recorded, and a top-level data addition record is generated;

[0028] S402: using the top-level data to add records, sorting the new data according to the time stamp, checking the priority access of the new data, integrating the new and old data using the insertion sort method, verifying the time sequence, and generating the new data time sorting result;

[0029] S403: Based on the time sorting result of the new data, simulate the stratigraphic superposition process, store the new data in a time-ordered manner at the top level of the data set, check the timeliness and access efficiency of the data structure, and generate a data hierarchy update result.

[0030] As a further solution of the present invention, the steps of compressing the data of the lower layer in the data hierarchy update result, simulating the compression density change during the geological layer compaction process, adjusting the storage parameters, and generating the formation compaction simulation result are specifically as follows:

[0031] S501: selecting lower-level data from the data hierarchy update result, evaluating compression potential, identifying and marking data that needs to be compressed, and adjusting storage density to generate a lower-level data selection result;

[0032] S502: According to the lower layer data selection result, use the Huffman coding algorithm to perform data compression on the selected data, monitor the changes of data integrity and access efficiency during the compression process, and generate compression parameter setting results;

[0033] S503: Using the compression parameter setting result, simulate the compression density change during the geological layer compaction process, adjust the storage space to match the compressed data density, optimize data storage efficiency and cost, and generate a formation compaction simulation result.

[0034] As a further solution of the present invention, the formula of the Huffman coding algorithm is as follows: in, is the compression efficiency value, Represents a data block The weight of Represents a data block The weight of Represents a data block The original length of Represents a data block The original length of represent and The absolute value difference.

[0035] As a further solution of the present invention, according to the formation compaction simulation results, combined with the data access frequency, the storage level of the data is dynamically adjusted, and the data access path is updated to match the change of the access mode. The steps of generating a dynamic access plan are specifically as follows:

[0036] S601: Based on the formation compaction simulation result, perform data hierarchy analysis, identify data with low storage density, archive the data, and prioritize high-frequency access data to obtain a data classification table;

[0037] S602: Using the data classification table, perform storage level measurement on the data, adjust low-frequency data to a lower-cost storage medium based on the measurement result, and migrate high-frequency data to a fast access level, optimize the storage structure, and generate a storage level adjustment record;

[0038] S603: Update the data access path through the storage level adjustment record, verify the conformity of the new path with the real-time access frequency, configure access rules and path pointing, optimize access efficiency, and generate a dynamic access plan.

[0039] A data storage system based on a large model, the data storage system based on a large model is used to execute the above-mentioned data storage method based on a large model, the system comprising:

[0040] The data collection and analysis module extracts data sets, identifies each data item and records the frequency of occurrence of the data, calculates the entropy value of the data to evaluate diversity and variability, and obtains entropy statistical results;

[0041] The data sorting and stratification module sorts the data items by entropy value based on the entropy value statistical results, locates the high entropy value data to the high-density storage area, and locates the low entropy value data to the low-density storage area, and constructs a hierarchical index structure;

[0042] The storage adjustment module uses the hierarchical index structure to analyze the data access frequency of each storage area, adjusts the physical location of the storage according to the data access frequency and entropy value, optimizes the access efficiency and storage cost, and forms a storage optimization plan;

[0043] The data compaction simulation module adds the newly collected data to the top layer of the data set according to the storage optimization scheme, and the new data is sorted according to the time stamp and stored in the top layer, simulating the stratigraphic superposition process, compressing the old data, and forming a hierarchical update and compaction result;

[0044] The dynamic storage management module dynamically adjusts the data storage level based on the hierarchical update and compaction results and changes in data access frequency, updates the data access path to match the changes in the access mode, and generates a dynamic access plan.

[0045] Compared with the prior art, the advantages and positive effects of the present invention are:

[0046] In the present invention, by collecting diverse data in a data set and calculating the number of occurrences and entropy values, the volatility and diversity of the data are evaluated, which can significantly improve data storage efficiency and access speed. The data is sorted and stored by entropy value, so that high entropy data is located in a high-density storage area, while low entropy data is stored in a low-density area, optimizing the utilization of storage space and reducing unnecessary storage costs. The data access frequency is calculated based on the data storage hierarchical index, and the physical storage location of the data is adjusted accordingly, reducing the data access time and improving the data processing speed. The hierarchical storage of newly added data and the sorting by time mark simulate the stratum superposition, which not only optimizes the storage structure, but also facilitates the rapid positioning and access of new data. The adjustment of data compression and storage parameters simulates the compaction process of geological layers, effectively manages storage capacity, and enhances the energy efficiency of data centers. Dynamically adjust the data storage hierarchy and update the access path to match the changes in access patterns, ensure the flexibility and adaptability of the storage system, and enhance the responsiveness to complex data environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the workflow of the present invention;

[0048] Figure 2 This is a detailed flow chart of S1 of the present invention;

[0049] Figure 3 This is a detailed flow chart of S2 of the present invention;

[0050] Figure 4 This is a detailed flow chart of S3 of the present invention;

[0051] Figure 5 This is a detailed flow chart of S4 of the present invention;

[0052] Figure 6 This is a detailed flow chart of S5 of the present invention;

[0053] Figure 7 This is a detailed flow chart of S6 of the present invention;

[0054] Figure 8 It is a system flow chart of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0056] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0057] See also Figure 1 The present invention provides a technical solution, a data storage method based on a large model, comprising the following steps:

[0058] S1: Collect multiple data in the large model data set, record the number of occurrences of each data one by one, use the frequency data to calculate the entropy value of each data, determine the volatility and diversity of the data, and generate the entropy value calculation result;

[0059] S2: Based on the entropy calculation results, use the large model to sort multiple data in the data set according to the entropy value, locate the data items with higher entropy values ​​in the high-density storage area, and locate the data items with lower entropy values ​​in the low-density storage area, and generate a hierarchical index for data storage;

[0060] S3: Based on the data storage hierarchical index, the data access frequency of multiple storage areas is calculated, and the data storage location is adjusted according to the frequency and entropy value, the physical storage location and access path of the data are optimized, the data retrieval efficiency is optimized, and the storage structure optimization plan is generated;

[0061] S4: Using the storage structure optimization scheme, the newly collected data are sequentially added to the top layer of the large model data set, sorted according to the time stamp of the new data, verified that the new data is stored in the top layer first, simulated the stratigraphic superposition process, and generated data hierarchy update results;

[0062] S5: compress the lower layer data in the data hierarchy update result, simulate the compression density change during the geological layer compaction process, adjust the storage parameters to reduce the storage space occupation, and generate the formation compaction simulation result;

[0063] S6: According to the results of the formation compaction simulation, the priority is adjusted in combination with the data access frequency, the storage level of the data is dynamically adjusted, the data access path is updated to match the changes in the access mode, the response speed and efficiency of the data access are checked, and a dynamic access plan is generated.

[0064] The entropy calculation results include a list of entropy values ​​of data items, identifiers of data items, corresponding entropy values ​​and rankings of entropy values. The data storage hierarchical index includes the sorting index of data items, the storage level corresponding to the data items and the storage density information of each level. The storage structure optimization plan includes the new configuration of the storage location, the comparison of access frequencies before and after the configuration and the capacity change of the storage area. The data hierarchy update results include the access time of the newly added data items, the storage level and the order of the data in the level. The formation compaction simulation results include the storage capacity before and after data compression, the compression rate and the access efficiency after compression. The dynamic access plan includes the adjustment record of the data level, the access path after the adjustment and the comparison of the response time before and after the path optimization.

[0065] See also Figure 2 , collect multiple data in the large model data set, record the number of occurrences of each data, calculate the entropy value of each data, determine the volatility and diversity of the data, and generate the entropy calculation results in the following steps:

[0066] S101: extract multiple data types from the large model data set, count the frequency of occurrence of each data type, store them in a dictionary in a key-value pair format, and obtain the data type frequency table. The execution process is as follows;

[0067] The process of extracting various types of data from a large model data set involves multiple types of data identification and classification. By scanning the data set, data types such as text, images, audio, etc. are identified and classified. Each time data is identified, the data type must be matched and recorded with the instance in which it occurs. In the process, each occurrence of data is recorded in a dictionary as a key-value pair, where the key is the data type and the value is the number of occurrences of the type of data. This not only improves the efficiency of data processing, but also facilitates subsequent data analysis operations. After scanning and recording all the data, a data type frequency table is obtained.

[0068] S102: Based on the data type frequency table, the proportion of the total number of occurrences of each data type to the total amount of the data set is calculated to obtain the data probability distribution table. The execution flow is as follows;

[0069] Calculate the proportion of the total number of occurrences of each data type to the total data set, according to the formula Calculate the probability distribution of the data type. In the formula, Represents data type The probability of Represents data type The number of occurrences of Represents the total amount of data in the data set. Detailed explanation of the formula and the process of formula calculation:

[0070] Set in the data set data items, data types Number of occurrences , then the data type Probability The calculation is as follows: This calculation example shows that the data type It accounts for 10% of the entire dataset.

[0071] S103: Using the data probability distribution table, the entropy value is calculated for each data type according to the probability value, and the negative value of the probability multiplied by the logarithm is cumulatively summed to obtain the entropy value calculation result. The execution process is as follows;

[0072] Calculating the entropy of a data type involves performing logarithmic calculations on the probability values ​​of each data type, determining the contribution of each type of data to the overall data entropy, and using the probability values ​​of the data types. and logarithm , multiply the two values ​​and take the negative value to get the entropy contribution value of the type data ,After this calculation is performed for all data types, all entropy contribution values ​​are accumulated to obtain the entropy value of the entire data set. The entropy value reflects the diversity and information richness of the data set. After the calculation is completed, it can be used to evaluate the complexity of the data set or determine the subsequent data processing strategy. The total entropy value is the sum of the entropy values ​​of all individual data types.

[0073] See also Figure 3 According to the entropy calculation results, the data is sorted by entropy using a large model, and data items with higher entropy values ​​are located in high-density storage areas, and data items with lower entropy values ​​are located in low-density storage areas. The specific steps for generating a hierarchical index for data storage are as follows:

[0074] S201: Based on the entropy calculation result, the data items are screened and classified, high entropy value and low entropy value data items are identified, a temporary identifier is assigned to each data item, marked as high priority and low priority, and the high entropy value data is prioritized to generate the entropy value grouping table. The execution process is as follows;

[0075] Filter and classify data items, and prioritize high entropy data according to the formula Generate an entropy value grouping table. In the formula, represents the sorted entropy value list, Represents data type The probability of represent The natural logarithm of . Detailed explanation of the formula and the process of formula calculation: Assume that the data type , , The probabilities are 0.1, 0.3, and 0.6 respectively, and the entropy values ​​are calculated as: , , . Sorting entropy value, get ,show is the highest entropy value, so in the entropy value grouping table At the front.

[0076] S202: Using the entropy grouping table, for high entropy data items, configure a storage area with read and write capabilities, migrate the data to the high-density storage area, perform the opposite operation on the low-entropy data items, allocate them to the low-density storage area with lower cost, update the storage configuration, and generate the execution process of the data allocation record as follows;

[0077] Based on the entropy grouping table, storage allocation is performed according to the entropy value of the data. High entropy data items require faster read and write speeds due to their rich information, so these data items are migrated to high-density storage areas with high read and write capabilities. Low entropy data items can be allocated to lower-cost, lower-density storage areas due to their lower information value. Data allocation not only optimizes the utilization efficiency of storage resources, but also reduces costs. It can effectively distinguish the storage location of data, ensure that high-value data obtains sufficient processing priority, improve the efficiency of data processing and access as a whole, and generate data allocation records.

[0078] S203: The execution process of generating a data storage hierarchical index by recording data allocation, synchronizing the access rights and speed settings of the storage area, checking the storage location of each data item and the entropy value match, configuring the data access path, optimizing the data access efficiency, and generating the data storage hierarchical index is as follows;

[0079] By checking the matching degree between the storage location and entropy value of each data item and configuring the data access path, we can ensure that each type of data gets the corresponding storage priority and access speed setting according to the information value. For example, high entropy data items allocated in high-speed storage areas will have faster access rights, while low entropy data items will have the opposite. This method significantly improves data access efficiency and reduces data access latency by optimizing data access paths, realizes reasonable hierarchical data storage, provides more refined management tools for data center operations, improves the overall performance of data services, and obtains hierarchical indexes for data storage.

[0080] See also Figure 4,Based on the data storage hierarchical index, the data access frequency of the storage area is calculated, the data storage location is adjusted according to the frequency and entropy value, and the physical storage location of the data is optimized. The specific steps of generating the storage structure optimization plan are as follows:

[0081] S301: Based on the data storage hierarchical index, the data access records of multiple storage areas are monitored, the access frequency of data items is calculated, and the execution process of generating data access frequency records is as follows;

[0082] Based on the hierarchical index of data storage, the data access records of multiple storage areas are monitored. The access status of data in each storage area is continuously tracked through a specially designed monitoring system, from which the access frequency of each data item is calculated and the number of accesses of each data item is recorded, including all data items and the corresponding access frequencies. This not only reflects the activity level of the data, but also provides a basis for subsequent storage optimization. It can clearly see which data is frequently accessed and which is less frequently accessed, and further analyze the data access pattern and user behavior, laying the foundation for optimizing the data storage structure and improving data access efficiency, and generating data access frequency records.

[0083] S302: Using data access frequency records, combined with the entropy value grouping table, comparing access frequency and entropy value, identifying data items with high access frequency and low entropy value, data items with low access frequency and high entropy value, adjusting storage locations, and generating a list of data items to be optimized. The execution process is as follows;

[0084] Compare the access frequency and entropy value to identify the data items that need to optimize the storage location, according to the formula , , generate a list of data items to be optimized. In the formula, Represents a list of data items to be optimized. The function is to filter out the frequency points that meet specific conditions. Represents data item The frequency of visits, Represents data item The entropy value of and Represents the threshold of access frequency and entropy value. Formula explanation and formula calculation derivation process: Setting access frequency threshold Times / month, entropy threshold , if the data item Frequency of visits And the entropy value ,but The storage location needs to be adjusted because the access frequency is high but the entropy value is low. and ,but They also need to be optimized because of their high entropy value and low access frequency. These two types of data items will be included in the list to be optimized middle.

[0085] S303: According to the list of data items to be optimized, reallocate data storage locations, move data with high access frequency to high-speed storage areas, move data with low access frequency to low-speed storage areas, optimize the physical storage structure of data, and generate the execution flow of the storage structure optimization solution as follows;

[0086] According to the list of data items to be optimized, the reallocation of data storage locations is achieved by migrating data items with high access frequency to high-speed storage areas and migrating data items with low access frequency to low-speed storage areas. This method of dynamically adjusting storage locations can significantly improve the efficiency of data access and optimize the overall storage cost. Through this method, not only the response speed to high-frequency access data is improved, but also the storage cost of infrequently accessed data is reduced. The strategy can also be continuously adjusted according to the actual access pattern to ensure that the data storage structure always maintains the optimal configuration and generates a storage structure optimization plan.

[0087] See also Figure 5 , using the storage structure optimization scheme, the newly collected data are sequentially added to the top layer of the large model data set, sorted according to the time stamp of the new data, and the new data are stored in the top layer first, simulating the stratigraphic superposition process. The specific steps for generating data hierarchy update results are as follows:

[0088] S401: According to the storage structure optimization plan, set the strategy for adding data to the large model data set, add the newly collected data to the top layer of the data set, record the time stamp of the new data, and generate the execution flow of the top layer data addition record as follows;

[0089] According to the storage structure optimization plan, the set data joining strategy ensures that all newly collected data are added to the top level of the large model data set. Each data item is attached with a time stamp when it is added to record the specific time of entry into the database. This not only simplifies the data tracking process, but also improves the queryability of the data, including the specific content of the data item and the corresponding time stamp. The records help managers quickly understand the update dynamics of the data, and at the same time provide an important basis for the subsequent data integration and analysis, ensuring that new data can be effectively integrated into the existing data structure, maintaining the integrity and up-to-dateness of the data set, and generating top-level data joining records.

[0090] S402: Use the top-level data to add records, sort the new data according to the time stamp, check the priority access of the new data, use the insertion sort method to integrate the new and old data, verify the time sequence, and generate the execution flow of the new data time sort result as follows;

[0091] The execution process of adding records with top-level data and sorting new data according to time stamps involves using the insertion sort method to integrate old and new data, and checking the priority access of new data to ensure the time sequence of data. The key lies in the time stamp of the new data, which provides the necessary reference information for data sorting. The insertion sort method inserts the new data into the existing data queue in sequence by comparing the time stamps of the data records. Although this method is not optimal when the amount of data is large, it can effectively maintain the time sequence of data when the amount of data is controllable. Verifying the time sequence is achieved by checking whether the data records are correctly sorted according to the time stamp one by one, ensuring that the position of each data is correct and generating the time sorting result of the new data.

[0092] S403: Based on the time sorting result of the new data, simulate the stratigraphic superposition process, store the new data in a time sequence on the top layer of the data set, check the timeliness and access efficiency of the data structure, and generate the data hierarchy update result. The execution process is as follows;

[0093] Based on the time sorting results of new data, the new data is stored in a simulated process at the top level of the data set in chronological order. The process verifies the timeliness and access efficiency of data items layer by layer to ensure that the integration of new data will not destroy the structure and logic of the original data. This layered storage method imitates the principle of stratigraphic superposition in geology. Each data layer represents a data snapshot of a time period, which not only increases the logic of the data structure, but also optimizes the data access path and improves access efficiency. It provides data administrators with a clear view of how data evolves over time. It is a key step to ensure continuous updating and effective management of data and generates data hierarchy update results.

[0094] See also Figure 6 , compress the lower layer data in the data hierarchy update result, simulate the compression density change during the geological layer compaction process, adjust the storage parameters, and generate the formation compaction simulation results in the following steps:

[0095] S501: Select lower-level data from the data hierarchy update result, evaluate compression potential, identify and mark data that needs to be compressed, and adjust storage density. The execution process of generating the lower-level data selection result is as follows;

[0096] When selecting lower-level data from the data hierarchy update results, it is necessary to conduct a detailed check on the current storage status of each data item, evaluate the compression potential through the data storage density and utilization, identify and mark the data items with high data redundancy and low access frequency as data that need to be compressed, and adjust the storage density of the marked data items to release storage space. This not only improves storage utilization, but also reduces storage maintenance costs. Recording the compression potential of each data item and the adjusted storage density provides a clear basis and guidance for the next compression process, and generates lower-level data selection results.

[0097] S502: According to the lower layer data selection result, the Huffman coding algorithm is used to perform data compression on the selected data, and the changes in data integrity and access efficiency during the compression process are monitored. The execution process of generating the compression parameter setting result is as follows;

[0098] The formula of Huffman coding algorithm is as follows: in, is the compression efficiency value, Represents a data block The weight of Represents a data block The weight of Represents a data block The original length of Represents a data block The original length of represent and The absolute value difference.

[0099] Detailed explanation of the formula and the process of formula calculation and derivation:

[0100] The formula is used to calculate the compression strategy for two data blocks, taking into account the size and importance of the data blocks to optimize the compression efficiency. and are the weights of the two data blocks, representing their importance in the overall data. The data block with higher weight should be compressed first. and Indicates the original length of the corresponding data block, that is, the data size. The denominator in the formula The influence of weight difference is taken into account, so that the compression calculation remains stable when the weight difference is large.

[0101] Setting data blocks Weight , data block Weight These weights are calculated based on the frequency of data access, where frequently accessed data blocks are given higher weights. Length , data block Length , these lengths are obtained through the storage information of the file system.

[0102] The specific calculation process is as follows:

[0103] Calculate the absolute value of the weight difference plus the square root of 1, that is, .

[0104] Calculating molecules .

[0105] Apply the formula to calculate the compression efficiency value .

[0106] The results show that the calculated compression efficiency value is 1506.85 considering the weight and length of the data block. This value represents the estimated optimal compression efficiency obtained by the Huffman coding algorithm under the given weight and data block size. This efficiency value reflects the efficiency of the compression process and will be used to evaluate whether to continue to adjust the compression parameters or optimize the compression strategy to further improve the performance of the storage system.

[0107] S503: Using the compression parameter setting results, simulating the compression density change during the geological layer compaction process, adjusting the storage space to match the compressed data density, optimizing data storage efficiency and cost, and generating the formation compaction simulation results. The execution process is as follows;

[0108] Simulate the changes in compression density during the formation compaction process, and adjust the storage space according to the formula to match the compressed data density. , where Represents the storage density after compression, Represents the storage density before compression, Represents the compression factor. Detailed explanation of the formula and the process of formula calculation: Setting the initial storage density , compression factor , the compressed storage density is calculated as follows: Calculations show that the data storage density after compression is 8GB / m³, matching the compressed data volume while optimizing data storage efficiency and reducing storage costs.

[0109] See also Figure 7 ,According to the results of stratum compaction simulation, combined with the data access frequency, the data storage level is dynamically adjusted, and the data access path is updated to match the changes in the access pattern. The steps for generating a dynamic access plan are as follows:

[0110] S601: Based on the formation compaction simulation results, perform data hierarchy analysis, identify data with low storage density, archive the data, and prioritize the high-frequency access data to obtain the data classification table. The execution process is as follows;

[0111] Based on the results of stratum compaction simulation, data with low storage density are identified through data hierarchy analysis, and archived. At the same time, data that is frequently accessed is prioritized to improve data access efficiency. The stratum compaction simulation results provide physical and logical density information of the data. Data hierarchy analysis uses this information to determine the optimization direction of data storage. Data with low storage density will not be accessed frequently, but will occupy a large physical space. Archiving this data can save storage costs. Prioritization of high-frequency access data is based on access frequency and data importance. Data is classified according to access frequency and storage density to obtain a data classification table.

[0112] S602: Using the data classification table, perform storage level measurement on the data, adjust low-frequency data to a lower-cost storage medium based on the measurement results, and migrate high-frequency data to a fast access level, optimize the storage structure, and generate a storage level adjustment record. The execution process is as follows;

[0113] The data classification table is used to measure the storage level of data. The measurement results are analyzed to adjust the low-frequency data to lower-cost storage media and migrate the high-frequency data to the fast-access level. This operation helps to optimize the entire storage structure. The data classification table provides the storage system with data stratification information. The storage level measurement is implemented based on this table. The adjustment of low-frequency data takes cost-effectiveness into consideration and selects low-cost but slow data recovery media. High-frequency data requires fast access, so it is migrated to a storage level with better performance and generates a storage level adjustment record.

[0114] S603: Update the data access path through storage level adjustment records, verify the conformity of the new path with the real-time access frequency, configure access rules and path pointing, optimize access efficiency, and generate the execution process of the dynamic access plan as follows;

[0115] Through the storage hierarchy adjustment records, the data access path is updated, the conformity of the new path with the real-time access frequency is verified, and the access rules and path pointing are configured. The detailed execution of the process includes the establishment of new access paths and the abolition of old paths. The verification work ensures that the setting of new paths meets the actual needs of data access. Through the analysis of the storage hierarchy adjustment records, when updating the access path, the access frequency and access delay of the data are focused on. When configuring the access rules, the requirements of data security and access control are taken into consideration. The adjustment of the path pointing is based on the physical location of the data and the expected access mode. The optimized access efficiency is reflected in the shortening of data access time and the increase of access frequency, and a dynamic access plan is generated.

[0116] See also Figure 8A data storage system based on a large model is provided, and the data storage system based on a large model is used to execute the data storage method based on a large model, and the system includes:

[0117] The data collection and analysis module extracts data sets, identifies each data item and records the frequency of occurrence of the data, calculates the entropy value of the data to evaluate diversity and variability, and obtains entropy statistical results;

[0118] The data sorting and stratification module sorts the data items by entropy value based on the entropy value statistics, locates the high entropy value data to the high-density storage area, and locates the low entropy value data to the low-density storage area, and builds a hierarchical index structure;

[0119] The storage adjustment module uses a hierarchical index structure to analyze the data access frequency of each storage area, adjusts the physical location of the storage according to the data access frequency and entropy value, optimizes the access efficiency and storage cost, and forms a storage optimization plan;

[0120] The data compaction simulation module adds the newly collected data to the top layer of the data set according to the storage optimization plan. The new data is sorted according to the time stamp and stored in the top layer, simulating the stratigraphic superposition process, compressing the old data, and forming a hierarchical update and compaction result;

[0121] The dynamic storage management module dynamically adjusts the data storage level based on the hierarchical update and compaction results and the changes in data access frequency, updates the data access path to match the changes in the access pattern, and generates a dynamic access plan.

[0122] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A data storage method based on a large model, characterized in that: The following steps are involved: Collect multiple data in the large model data set, record the number of occurrences of each data, calculate the entropy value of each data, determine the volatility and diversity of the data, and generate entropy calculation results; According to the entropy value calculation result, the data is sorted by entropy value using a large model, data items with higher entropy values ​​are located in a high-density storage area, and data items with lower entropy values ​​are located in a low-density storage area, and a hierarchical index for data storage is generated; Based on the data storage hierarchical index, the data access frequency of the storage area is calculated, the data storage location is adjusted according to the frequency and entropy value, the physical storage location of the data is optimized, and a storage structure optimization plan is generated; By using the storage structure optimization scheme, newly collected data are sequentially added to the top layer of the large model data set, sorted according to the time stamp of the new data, and the new data are preferentially stored in the top layer, simulating the stratigraphic superposition process to generate data hierarchy update results; Compressing the data of the lower layer in the data hierarchy update result, simulating the compression density change during the geological layer compaction process, adjusting the storage parameters, and generating the formation compaction simulation result; According to the formation compaction simulation results, combined with the data access frequency, the data storage level is dynamically adjusted, the data access path is updated to match the change of the access mode, and a dynamic access plan is generated; The entropy value calculation result includes a list of entropy values ​​of data items, identifiers of data items, corresponding entropy values ​​and rankings of entropy value sizes; the data storage hierarchical index includes a sorting index of data items, storage levels corresponding to data items and storage density information of each level; the storage structure optimization plan includes a new configuration of storage locations, a comparison of access frequencies before and after configuration and a capacity change of storage areas; the data hierarchy update result includes the access time of newly added data items, the storage level and the order of data in the level; the formation compaction simulation result includes storage capacity before and after data compression, compression rate and access efficiency after compression; the dynamic access plan includes adjustment records of data levels, access paths after adjustment and comparison of response times before and after path optimization.

2. The data storage method based on a large model according to claim 1, characterized in that: Collect multiple data in the large model data set, record the number of occurrences of each data, calculate the entropy value of each data, determine the volatility and diversity of the data, and generate the entropy calculation results in the following steps: Extract multiple data types from the large model data set, count the frequency of each data type, store them in a dictionary using key-value pairs, and obtain a data type frequency table; Based on the data type frequency table, calculate the proportion of the total number of occurrences of each data type to the total number of data sets to obtain a data probability distribution table; The data probability distribution table is used to calculate the entropy value for each data type according to the probability value, and the entropy value calculation result is obtained by accumulating and summing the probability multiplied by the negative value of the logarithm.

3. The data storage method based on a large model according to claim 1, characterized in that: According to the entropy value calculation result, the data is sorted by entropy value using a large model, and data items with higher entropy values ​​are located in a high-density storage area, and data items with lower entropy values ​​are located in a low-density storage area. The steps of generating a hierarchical index for data storage are specifically as follows: Based on the entropy value calculation result, the data items are screened and classified, high entropy value and low entropy value data items are identified, a temporary identifier is assigned to each data item, marked as high priority and low priority, and the high entropy value data is prioritized to generate an entropy value grouping table; Using the entropy grouping table, for high entropy data items, a storage area with strong read and write capabilities is configured, the data is migrated to the high-density storage area, and the opposite operation is performed on the low-entropy data items, which are allocated to the low-density storage area with lower cost, the storage configuration is updated, and a data allocation record is generated; Through the data allocation record, the access rights and speed settings of the storage area are synchronized, the storage location of each data item is checked to match the entropy value, the data access path is configured, the data access efficiency is optimized, and a data storage hierarchical index is generated.

4. The data storage method based on a large model according to claim 1, characterized in that: Based on the data storage hierarchical index, the data access frequency of the storage area is calculated, the data storage location is adjusted according to the frequency and entropy value, the physical storage location of the data is optimized, and the steps of generating a storage structure optimization solution are specifically as follows: Based on the data storage hierarchical index, monitoring data access records of multiple storage areas, calculating access frequencies of data items, and generating data access frequency records; Using the data access frequency record, combined with the entropy value grouping table, comparing the access frequency and the entropy value, identifying the data items with high access frequency and low entropy value, and the data items with low access frequency and high entropy value, adjusting the storage location, and generating a list of data items to be optimized; According to the list of data items to be optimized, data storage locations are reallocated, data with high access frequency is moved to a high-speed storage area, and data with low access frequency is moved to a low-speed storage area, the physical storage structure of the data is optimized, and a storage structure optimization plan is generated.

5. The data storage method based on a large model according to claim 1, characterized in that: By using the storage structure optimization scheme, the newly collected data is sequentially added to the top layer of the large model data set, sorted according to the time stamp of the new data, and the new data is preferentially stored in the top layer, simulating the stratigraphic superposition process, and generating the data hierarchy update results in the following specific steps: According to the storage structure optimization scheme, a strategy for adding data to the large model data set is set, the newly collected data is added to the top layer of the data set, the time stamp of the new data is recorded, and a top-level data addition record is generated; Using the top-level data to add records, sorting the new data according to the time stamp, checking the priority access of the new data, integrating the new and old data using the insertion sort method, verifying the time sequence, and generating the new data time sorting result; Based on the time sorting result of the new data, the stratigraphic superposition process is simulated, the new data is stacked and stored at the top level of the data set in chronological order, the timeliness and access efficiency of the data structure are checked, and the data hierarchy update result is generated.

6. The data storage method based on a large model according to claim 1, characterized in that: The steps of compressing the lower layer data in the data hierarchy update result, simulating the compression density change during the geological layer compaction process, adjusting the storage parameters, and generating the formation compaction simulation result are specifically as follows: Selecting lower-level data from the data hierarchy update results, evaluating compression potential, identifying and marking data that needs to be compressed, and adjusting storage density to generate lower-level data selection results; According to the lower layer data selection result, using Huffman coding algorithm, performing data compression on the selected data, monitoring the changes of data integrity and access efficiency during the compression process, and generating compression parameter setting results; The compression parameter setting result is used to simulate the compression density change during the geological layer compaction process, adjust the storage space to match the compressed data density, optimize data storage efficiency and cost, and generate a stratum compaction simulation result.

7. The data storage method based on a large model according to claim 6 is characterized in that: The formula of the Huffman coding algorithm is as follows: in, is the compression efficiency value, Represents a data block The weight of Represents a data block The weight of Represents a data block The original length of Represents a data block The original length of represent and The absolute value difference.

8. The data storage method based on a large model according to claim 1, characterized in that: According to the formation compaction simulation results, combined with the data access frequency, the data storage level is dynamically adjusted, and the data access path is updated to match the change of the access mode. The steps of generating a dynamic access plan are specifically as follows: Based on the formation compaction simulation results, data hierarchy analysis is performed to identify data with low storage density, archive the data, and prioritize high-frequency access data to obtain a data classification table; The data classification table is used to measure the storage level of the data, and low-frequency data is adjusted to a lower-cost storage medium according to the measurement result, and high-frequency data is migrated to a fast access level, the storage structure is optimized, and a storage level adjustment record is generated; Through the storage level adjustment record, the data access path is updated, the conformity of the new path with the real-time access frequency is verified, the access rules and path pointing are configured, the access efficiency is optimized, and a dynamic access plan is generated.

9. A data storage system based on a large model, characterized in that: According to the large model-based data storage method according to any one of claims 1 to 8, the system comprises: The data collection and analysis module extracts the data set, identifies each data item and records the frequency of occurrence of the data, calculates the entropy value of the data to evaluate the diversity and variability, and obtains the entropy value calculation result; The data sorting and stratification module performs entropy sorting on the data items based on the entropy value calculation result, locates the data items with higher entropy values ​​in the high-density storage area, and locates the data items with lower entropy values ​​in the low-density storage area, and constructs a data storage stratification index; The storage adjustment module uses the data storage hierarchical index to analyze the data access frequency of each storage area, adjusts the physical location of the storage according to the data access frequency and entropy value, optimizes the access efficiency and storage cost, and forms a storage structure optimization plan; The data compaction simulation module adds the newly collected data to the top layer of the large model data set according to the storage structure optimization scheme, sorts the new data according to the time stamp and stores it in the top layer, simulates the stratigraphic superposition process, compresses the old data, and forms a data hierarchy update result; The dynamic storage management module dynamically adjusts the data storage level based on the data level update result and in combination with the change of data access frequency, updates the data access path to match the change of access mode, and generates a dynamic access plan; The entropy value calculation result includes a list of entropy values ​​of data items, identifiers of data items, corresponding entropy values ​​and rankings of entropy value sizes; the data storage hierarchical index includes a sorting index of data items, storage levels corresponding to data items and storage density information of each level; the storage structure optimization plan includes a new configuration of storage locations, a comparison of access frequencies before and after configuration and a capacity change of storage areas; the data hierarchy update result includes the access time of newly added data items, the storage level and the order of data in the level; the formation compaction simulation result includes storage capacity before and after data compression, compression rate and access efficiency after compression; the dynamic access plan includes adjustment records of data levels, access paths after adjustment and comparison of response times before and after path optimization.

Images