Data storage method and medium of an AI all-in-one system

Abstract

The application relates to the technical field of data processing, in particular to a data storage method of an AI all-in-one machine system and a medium, which solves the technical problem of low data access efficiency caused by unreasonable data storage in the prior art. The method comprises the following steps: determining access and demand statistical characteristics of data blocks in different storage levels in the AI all-in-one machine system; the access and demand statistical characteristics comprise an access frequency of the data blocks in the present migration period, a time of the latest access to the data blocks in the present migration period, and a storage duration of the data blocks in the storage level; a corresponding cross-level data migration strategy is established by analyzing the access and demand statistical characteristics of the data blocks in different storage levels; the cross-level data migration strategy is used for representing target storage levels of the data blocks in the next migration period; and the storage levels of the data blocks are dynamically updated based on the cross-level data migration strategy when the migration period is switched.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data storage method of an AI all-in-one machine system and a medium. BACKGROUND

[0002] An artificial intelligence (AI) all-in-one machine system is a device or a group of devices that packages hardware and software required for model training, fine-tuning, reasoning, data collection / preprocessing, and operation and maintenance into a deliverable and easily deployable device. The goal is to be "ready to use, controllable, safe, performance measurable, and easy to integrate". It is generally used for private reasoning and data edge-cloud collaboration in enterprises, which can reduce integration complexity, achieve higher throughput or lower latency, and reduce operation and maintenance threshold. At the same time, private deployment can meet data privacy and compliance requirements.

[0003] The current data storage system of the AI all-in-one machine generally adopts a cold / hot layered storage architecture. Hot data with high access frequency is stored in the hot layer (high-performance storage medium), and cold data with infrequent access is stored in the cold layer (general storage medium). However, due to the deployment and use of the all-in-one machine, massive data access is accompanied by dynamic changes in the hot / cold types of some data at different time nodes. Fixed data storage levels (such as hot data changing to cold data but still stored in the hot layer) can reduce overall performance and affect the efficiency of model training. SUMMARY

[0004] In order to solve the technical problem of low data access efficiency caused by unreasonable data storage in the prior art, the purpose of the present application is to provide a data storage method of an AI all-in-one machine system. The technical solution adopted is as follows:

[0005] Determine the access and demand statistical characteristics of data blocks in different storage levels in the AI all-in-one machine system. The access and demand statistical characteristics include the access frequency of the data block in the current migration period, the time of the last access to the data block in the current migration period, and the storage duration of the data block in the storage level.

[0006] Establish a cross-level data migration strategy based on the analysis of the access and demand statistical characteristics of the data blocks in different storage levels. The cross-level data migration strategy is used to represent the target storage level of each data block in the next migration period.

[0007] Based on the cross-level data migration strategy, dynamically update the storage level of each data block at the migration period switching.

[0008] In a possible implementation, the method comprises: for each data block, calculating a hotness value of the data block at the end of the current migration period according to the access frequency corresponding to the data block, the time of the last access to the data block in the current migration period, and the storage duration at the storage level where the data block is located;

[0009] Based on the hotness value of each data block and the hotness average value of each storage level, an abnormal data block set in each storage level is screened out;

[0010] For each abnormal data block set, the migration direction of the data blocks in the abnormal data block set is determined, and the migration matching degree of the data blocks in the abnormal data block set and each candidate storage level is calculated; the candidate storage level is the storage level in the migration direction of the corresponding data block;

[0011] According to the migration matching degree, the target storage level of the data block in the next migration period is determined to establish a cross-level data migration strategy.

[0012] In a possible implementation, the method comprises: determining the hotness abnormality degree of each data block based on the hotness value of each data block and the hotness average value of each storage level, and clustering according to the deviation between the hotness abnormality degrees of different data blocks to screen out an abnormal data block set in each storage level.

[0013] In a possible implementation, the migration direction is determined based on the hotness value of the data block in the last migration period and the hotness value in the current migration period;

[0014] The migration matching degree of the data block and the candidate storage level is determined based on the hotness value of the data block in the current migration period, the hotness average value of the candidate storage level, and the data size of the data in the candidate storage level belonging to the same data file as the data block.

[0015] In a possible implementation, in the case where there is no storage level in the migration direction of the data block, the target storage level of the data block in the next migration period is the storage level where the data block is located in the current migration period.

[0016] In a possible implementation, the method further comprises:

[0017] According to the sampling interval, the data information of the real-time input data input into the AI all-in-one machine system in real time is obtained, the generation delay and the average compression rate are generated;

[0018] Based on the data information of the real-time input data and the generation delay, a correlation parameter of the real-time input data and the data obtained in the sampling interval at the previous moment is determined, and based on the correlation parameter and the average compression rate, a storage priority of the real-time input data is determined; the storage priority is used to determine the initial storage level of the real-time input data;

[0019] According to the storage priority, the real-time input data is stored to the corresponding storage level.

[0020] In a possible implementation, the value range of the storage priority is divided into multiple intervals; the multiple intervals respectively correspond to the storage levels in the AI all-in-one machine system in a one-to-one manner, and the initial storage level of the real-time input data is the storage level corresponding to the interval in which the storage priority of the real-time input data is located.

[0021] In a possible implementation, the storage levels in the AI all-in-one machine system are divided in the order from high to low according to the data reading performance.

[0022] In a possible implementation, the storage levels in the AI all-in-one machine system include a hot data layer, a warm data layer, and a cold data layer; the data reading performance of the hot data layer is greater than that of the warm data layer; and the data reading performance of the warm data layer is greater than that of the cold data layer.

[0023] The present application provides a computer readable storage medium, the computer readable storage medium stores instructions, when a computer executes the instructions, the computer executes the method described in any of the above embodiments.

[0024] The present application has the following beneficial effects:

[0025] The present application provides a data storage method of an AI all-in-one machine system, by determining the access and demand statistical characteristics (such as access frequency, recent access time, and storage duration) of each storage level data block, and establishing a cross-level data migration strategy by analyzing the statistical characteristics, the target storage level of the data block is determined, then the storage level is dynamically updated when the migration period is switched, the dynamic adaptation of the data storage level and the data hot / cold type is realized. Therefore, the present application effectively solves the low performance problem caused by the fixed data storage level in the existing cold / hot layered storage architecture, ensures that the hot data is always stored in the high-performance medium and the cold data is stored in the low-cost medium, thereby improving the data access efficiency of the AI all-in-one machine system and reducing the device running delay. BRIEF DESCRIPTION OF DRAWINGS

[0026] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0027] Figure 1 A flowchart of a data storage method of an AI all-in-one machine system provided by an embodiment of the present application;

[0028] Figure 2 A structure diagram of a storage hierarchy of an AI all-in-one machine system provided by one embodiment of the present application;

[0029] Figure 3 A flowchart of a data storage method of an AI all-in-one machine system provided by one embodiment of the present application;

[0030] Figure 4 A flowchart of a data storage method of an AI all-in-one machine system provided by one embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined purposes, the following describes in detail the specific embodiments, structures, features and effects of a data storage method and medium of an AI all-in-one machine system according to the present application, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures or characteristics in one or more embodiments can be combined in any suitable form.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0033] An artificial intelligence (AI) all-in-one machine system is a device or cluster that packages hardware and software required for model training, fine-tuning, reasoning, data collection / preprocessing and operation and maintenance into a deliverable and easily deployable device. The goal is "plug and play, controllable, safe, performance measurable, and easy to integrate". It is generally used for private reasoning and edge-cloud collaboration of data in enterprises, which can reduce integration complexity, obtain higher throughput or lower latency, and reduce operation and maintenance threshold. At the same time, private deployment can meet the needs of data privacy and compliance.

[0034] In the actual application of the AI all-in-one machine system, the data storage system is a key component supporting core tasks such as model training and reasoning. The current mainstream AI all-in-one machine data storage system generally adopts a cold / hot layered storage architecture, that is, by storing hot data with high access frequency in a high-performance storage medium (hot layer) and storing cold data that is not frequently accessed in a general storage medium (cold layer), the storage performance and cost are balanced.

[0035] However, due to the continuous access of massive data during the running of the AI all-in-one machine, the access frequency (i.e. hot and cold types) of the same data at different time nodes may dynamically change, and the existing fixed data storage hierarchy cannot adapt to such changes, resulting in the situation that part of the hot data is still stored in the low-performance storage medium, and part of the cold data occupies the high-performance storage medium, ultimately causing the overall performance of the AI all-in-one machine system to decline, the data access efficiency to decrease, and the efficiency of model training and inference tasks to be affected.

[0036] In view of this, the embodiments of the present application provide a data storage method of an AI all-in-one machine system, which determines the access and demand statistical characteristics (such as access frequency, recent access time, and storage duration) of the data blocks in each storage hierarchy, and establishes a cross-level data migration strategy by analyzing the statistical characteristics, so as to determine the target storage hierarchy of the data blocks. Then, by dynamically updating the storage hierarchy when the migration period is switched, the dynamic adaptation of the data storage hierarchy and the data hot and cold types is realized. Therefore, the present application effectively solves the problem of low performance caused by the fixed data storage hierarchy in the existing cold / hot layered storage architecture, ensures that the hot data is always stored in the high-performance medium and the cold data is stored in the low-cost medium, thereby improving the data access efficiency of the AI all-in-one machine system and reducing the device running delay.

[0037] The specific scheme of the data storage method of the AI all-in-one machine system provided by the present application will be described in detail below in conjunction with the drawings.

[0038] Please refer to Figure 1 which shows the method flowchart of the data storage method of the AI all-in-one machine system provided by an embodiment of the present application, which comprises the following steps:

[0039] Step 101, determining the access and demand statistical characteristics of the data blocks in different storage hierarchies in the AI all-in-one machine system.

[0040] Among them, the access and demand statistical characteristics include the access frequency of the data blocks in the current migration period, the time of the last access of the data blocks in the current migration period, and the storage duration of the data blocks in the storage hierarchy.

[0041] In some embodiments, the storage hierarchies in the AI all-in-one machine system are divided in order from high to low according to the data reading performance.

[0042] Among them, the AI all-in-one machine generally integrates the hardware and software required for model training, fine-tuning, inference, data acquisition / preprocessing, and operation and maintenance management, and the data storage device in the hardware part mainly adopts a hot / cold layered storage architecture, which can improve performance, reduce cost, and enhance the overall management efficiency and reliability of the system when facing large-scale data processing and model training and inference tasks.

[0043] For example, asFigure 2 As shown, the storage hierarchy in the AI all-in-one system is usually divided based on performance and cost, and the storage hierarchy in the AI all-in-one system includes a hot data layer, a warm data layer, and a cold data layer. The data read performance of the hot data layer is greater than that of the warm data layer, and the data read performance of the warm data layer is greater than that of the cold data layer.

[0044] Among them, the hot data layer is mainly used for frequently accessed data of training / online inference, and the storage hardware thereof mainly includes: high-performance non-volatile memory express solid state drive (NVMe SSD), Optane / Pmem, etc. The warm data layer is used to store data that is often read but does not require extremely low latency, and the storage hardware thereof is: capacity-optimized serial advanced technology attachment (SATA) serial attached small computer system interface (SAS) enterprise SSD or high-speed hard disk hard disk drive (HDD) + SSD cache. The cold data layer is used to store low-frequency access data or backup data, and the storage hardware thereof is low-cost large-capacity HDD (Just a Bunch of Disks, JBOD), distributed file system distributed storage system (Ceph reliable autonomic distributed object store, Ceph RADOS), etc.

[0045] Since the hot and cold types corresponding to the same data at different time nodes may dynamically change, if the data is not migrated and updated, there may be a case that hot data is stored in a low-performance storage medium. Since the access frequency of hot data is usually high, the performance of the low-performance storage medium is difficult to meet the efficient data read demand, thereby causing a performance bottleneck. Similarly, storing cold data in a high-performance storage medium will waste storage resources, causing cost or performance imbalance. Therefore, the present application can dynamically migrate data blocks in real time by collecting access and demand statistical characteristics to improve overall efficiency.

[0046] In some implementations, the present application can obtain access and demand statistical characteristics through a detection module. The sampling interval of the detection module can be set as needed, for example, it can be set to 1 ms, to ensure the real-time and accuracy of the statistical characteristics.

[0047] In some embodiments, the data block is the basic unit of data storage in the AI all-in-one system. For example, a data file can be divided into multiple data blocks for storage, facilitating subsequent independent migration and management. Migration period refers to the time interval of data level migration in the AI all-in-one system. Frequent data migration increases additional cost and performance, so the migration period needs to be determined in combination with the hardware performance of the AI all-in-one system and the business difficulty to achieve the optimal balance of performance and energy consumption. For example, the migration period in the present embodiment can be set to 3 days, i.e., the data blocks in each storage level are migrated once every 3 days for migration analysis and level update. In some embodiments, if the AI all-in-one machine runs high-frequency inference tasks, the migration period can be shortened to 1 day, and if it runs low-frequency training tasks, the migration period can be extended to 7 days.

[0048] Step 102: Establish a cross-level data migration strategy by analyzing the access and demand statistical characteristics of the data blocks in different storage levels.

[0049] The cross-level data migration strategy is used to represent the target storage level of each data block in the next migration period.

[0050] It should be noted that the cross-level data migration strategy can be represented by a rule set that guides the migration of data blocks between different storage levels. The core is to determine whether the data block needs to be migrated and to which target storage level according to the access and demand statistical characteristics of the data block.

[0051] As the inference task changes and the business expands, the data hotness of different levels may no longer be suitable for the corresponding level, so if the data is not updated by level migration, it will affect the running efficiency of the all-in-one machine. Therefore, the present application can establish a cross-level data migration strategy based on the differences in the access and demand statistical characteristics of the data blocks in different storage levels.

[0052] For example, in combination with Figure 2 The storage architecture described in the present embodiment is divided into three levels. The performance of the storage medium of each level from top to bottom is lower and the data hotness gradually decreases. If the hotness of some data blocks changes, they need to be migrated to the level that meets the hotness. The access characteristics of different data blocks in each migration period represent their hotness performance.

[0053] For example, if the access frequency of a data block in the warm data layer significantly increases in the current migration period, its migration strategy can be determined as "migrate to the hot data layer in the next migration period"; if the access frequency of a data block in the hot data layer significantly decreases, its migration strategy can be determined as "migrate to the warm data layer in the next migration period".

[0054] Step 103, dynamically updating the storage level of each data block at the migration period switching based on the cross-level data migration strategy.

[0055] The migration period switching refers to the time node of the end of the current migration period and the start of the next migration period. The application can perform cross-level migration on the data blocks stored in the AI all-in-one machine system at each migration period through the above-mentioned manner.

[0056] It should be noted that the manner described in the embodiments of the application only represents the strategy of the migration process. After the migration process ends, the next migration period is entered, and the heat of the data is continuously detected in the next migration period, and the data migration is performed based on the strategy at the end of the period, and so on for dynamic updating. In some embodiments, the migration operation of the data block needs to avoid the peak period of the AI all-in-one machine (such as the period of the model training task set) to avoid occupying too many system resources in the migration process and affecting the operation of the core business.

[0057] Based on the above technical solution, the application determines the access and demand statistical characteristics (such as access frequency, recent access time, and storage duration) of each storage level data block, and establishes a cross-level data migration strategy by analyzing the statistical characteristics, thereby determining the target storage level of the data block. Then, the storage level is dynamically updated at the migration period switching, which realizes the dynamic adaptation of the data storage level and the data hot / cold type. Therefore, the application effectively solves the low performance problem caused by the fixed data storage level in the existing cold / hot layered storage architecture, ensures that the hot data is always stored in the high-performance medium and the cold data is stored in the low-cost medium, thereby improving the data access efficiency of the AI all-in-one machine system and reducing the device running delay.

[0058] As a possible embodiment of the application, in combination with Figure 1 As shown in Figure 3 The above step 102 can be implemented through the following steps:

[0059] Step 301, for each data block, according to the access frequency corresponding to the data block, the time of the last access to the data block in the current migration period, and the storage duration in the storage level, the heat value of the data block at the end of the current migration period is calculated.

[0060] The heat value is a parameter for quantifying the access demand degree of the data block. The higher the heat value, the more vigorous the access demand of the data block, and the more suitable it is to be stored in the high-performance storage level.

[0061] In some embodiments, the heat value satisfies the following formula:

[0062]

[0063] wherein, is the data block the hotness value at the end of the current migration cycle. is the data block the access frequency corresponding to the current migration cycle, is the time of the most recent access to this data block within the current migration cycle, is the time at the end of the current migration cycle. is the storage duration of this data block at the storage level.

[0064] That is, the above represent the access characteristics of this data block . If the access frequency of this data block is higher in the current migration cycle, and the most recent access is closer to the time at the end of the current migration cycle, it indicates that the demand for this data block is higher, and the storage duration is used as a weight, that is, the longer the storage duration, the greater the possibility of migration, and the lower the corresponding hotness.

[0065] Step 302, based on the hotness value of each data block and the hotness average of each storage level, filtering out an abnormal data block set in each storage level.

[0066] Among them, the hotness average of the storage level refers to the average value of the hotness values of all data blocks in the storage level, which is used to represent the overall access demand level of the data in the level; the abnormal data block set refers to the set of data blocks whose hotness values deviate greatly from the hotness average of the level and are not suitable for continuing to be stored in the current storage level.

[0067] Since different storage levels correspond to different data access hotness and the hotness of the data blocks changes, and the purpose of data migration is to migrate data blocks that do not meet the hotness to appropriate storage levels, the present application can filter data blocks of different storage levels based on hotness values to determine abnormal data blocks that need to be migrated.

[0068] For any storage level, if the hotness value of a certain data block in the current migration cycle deviates greatly from the hotness value of the overall data of the storage level, it indicates that the hotness of the data block is no longer suitable for the level and needs to be migrated.

[0069] In one possible implementation, the present application can determine the hotness abnormality of each data block based on the hotness value of each data block and the hotness average of each storage level, and cluster according to the deviation between the hotness abnormality of different data blocks to filter out an abnormal data block set in each storage level.

[0070] wherein the hotness abnormality degree is used to represent the deviation of the data block hotness from the level mean value. The clustering algorithm in the embodiments of the present application can calculate the clustering distance of different data blocks based on the hotness abnormality degree, so as to realize clustering. For example, the clustering algorithm includes K-means clustering, hierarchical clustering, DBSCAN clustering, etc.

[0071] In some embodiments, the hotness abnormality degree can be represented by the following formula:

[0072]

[0073] wherein, represents the hotness abnormality degree of the data block , is the hotness value of the data block at the end of the current migration period, is the hotness mean value of all data blocks in the storage level where the data block is located.

[0074] Therefore the value of the data block represents the deviation of the hotness value of the data block from the overall data block hotness value of the storage level where the data block is located. The greater the hotness abnormality degree, the more unsuitable the corresponding data block is for being stored in the storage level, and needs to be migrated out (for example, after normalization, the value corresponds to an abnormal data block).

[0075] In some embodiments, the deviation between the hotness abnormality degrees of different data blocks can be represented by the following formula:

[0076]

[0077] wherein, is the deviation (i.e. clustering distance) between the hotness abnormality degrees of the data block and the data block . represents the hotness abnormality degree of the data block , represents the hotness abnormality degree of the data block , is an infinitesimal positive number, which in the embodiments of the present application may also be 0.01.

[0078] The embodiment accurately quantifies the adaptation deviation of the data block and the level by calculating the heat abnormality degree of the data block, and provides a clear standard for the preliminary screening of the abnormal data block. The abnormal data block is clustered based on the clustering distance, and the data blocks with similar heat abnormality degrees are classified into the same set, so that the migration direction and the target level are determined in batches subsequently. The above scheme improves the efficiency of the abnormal data block screening and reduces the complexity of the system migration operation.

[0079] The heat abnormality degrees of different abnormal data blocks are compared. If the heat abnormality degrees of two data blocks are more similar, that is, smaller, the two data blocks are more likely to be classified into the same category. Thus, the abnormal data blocks can be classified and divided by combining the clustering method.

[0080] The data of each storage level is clustered and screened by the above-mentioned method, and a plurality of abnormal clustering sets can be obtained. The data block sets can be migrated at the end of each migration period. The migration direction and the target storage level of different clustering sets need to be determined.

[0081] Step 303: For each abnormal data block set, the migration direction of the data block in the abnormal data block set is determined, and the migration matching degree of the data block in the abnormal data block set and each candidate storage level is calculated.

[0082] The candidate storage level is the storage level in the migration direction of the corresponding data block.

[0083] For any abnormal data block set, the target is to migrate it to the storage level that best meets its heat value. The determination of the migration direction needs to compare the heat change of the data block. That is, if the heat of the data block in the abnormal clustering set in this period is larger than that in the last period, it means that the access demand of the data is larger, and the data needs to be migrated to the upper level, otherwise, the data needs to be migrated to the lower level.

[0084] In some embodiments, the migration direction is determined based on the heat value of the data block in the last migration period and the heat value in the current migration period.

[0085] For example, the migration direction can be represented by a migration direction coefficient, which can be represented by the following formula:

[0086]

[0087] wherein, is the migration direction coefficient of the data block . is the heat value of the data block at the end of the last migration period, is the heat value of the data block The hotness value at the end of the last migration cycle. If greater than 0 indicates that the hotness value of the data block is relatively rising, and the migration direction is upward migration, and vice versa less than 0 indicates that the hotness value of the data block is relatively falling, and the migration direction is downward migration. In other cases, i.e. equal to 0 or not present, it means that no hierarchical migration is performed.

[0088] In some embodiments, in the absence of a storage hierarchy in the migration direction of the data block, the target storage hierarchy of the data block in the next migration cycle is the storage hierarchy in which the data block is located in the current migration cycle. That is, if some abnormal data blocks have reached the uppermost layer or the lowermost layer, and still meet the conditions for upward or downward migration, hierarchical migration is still not performed.

[0089] In this way, the present application can determine the migration direction of the data block in the abnormal data block set corresponding to each storage hierarchy and mark all storage hierarchies in this direction as candidate storage hierarchies. Then, since different types of AI all-in-one machines correspond to different storage hierarchy architectures, there may be more storage hierarchies, so after determining the migration direction, the target storage hierarchy of the migration needs to be determined.

[0090] The target hierarchy of the migration corresponds to the landing position of the data block, and the data block needs to be stored in a medium that better meets its hotness. In addition, different data blocks may belong to different data files, so storing data blocks of the same data file in the same storage hierarchy as much as possible can improve the overall access efficiency of the system.

[0091] In some embodiments, the migration matching degree of the data block and the candidate storage hierarchy is determined based on the hotness value of the data block in the current migration cycle, the hotness mean of the candidate storage hierarchy, and the data size of the data blocks in the candidate storage hierarchy that belong to the same data file as the data block.

[0092] For example, the migration matching degree of the data block and the candidate storage hierarchy can be represented by the following formula:

[0093]

[0094] Wherein, is the migration matching degree of the data block and the candidate storage hierarchy . is the hotness value of the data block at the end of the current migration cycle, is the hotness mean of all data blocks in the candidate storage hierarchy . is the candidate storage hierarchy the data block the data size (e.g., the number of bytes) of the data belonging to the same data file, is a positive infinitesimal. That is to say, if a candidate storage level is similar to the hot degree of the data block , the smaller the value is, and the greater the belonging degree of the same data file is, the more necessary it is to migrate the data block to the storage level.

[0095] Step 304, determining the target storage level of the data block in the next migration period according to the migration matching degree, to establish a cross-level data migration strategy.

[0096] Among them, the target storage level refers to the level with the highest migration matching degree in the candidate storage level, and the application can match each data block in the abnormal data block set and select the candidate storage level with the largest matching degree as the target storage level for data migration, and update the index.

[0097] Based on the above technical solution, the application can quantify the access demand of the data block by calculating the hot degree value of the data block, providing a quantitative basis for abnormal data screening. Then, by screening the abnormal data block set based on the hot degree value and the level average, the application accurately identifies the data that is not suitable for the current level, and determines the migration direction and calculates the migration matching degree, thereby determining the migration trend and adaptive level of the data block. In this way, the application can determine the target storage level according to the matching degree, and finally establish a precise cross-level data migration strategy. The above scheme can ensure that the target storage level of each data block can match its actual access demand, avoid wasting system resources caused by blind migration, and improve the accuracy and efficiency of data migration.

[0098] In addition, for an AI all-in-one machine, its main role is to train large models and perform task reasoning (such as real-time reasoning of data streams and the like through a deep learning model, and making quick decisions, etc.), and these processes are accompanied by the input of real-time data. When these data are input in real time, they need to be stored in the corresponding level for reasoning access, and the appropriate data storage level determines the reasoning efficiency of the model. Therefore, the application can further determine the storage level of the real-time input all-in-one machine.

[0099] As a possible embodiment of the application, in combination with Figure 1 , as shown in FIG. 4, the method further includes the following steps: Figure 4 Step 401, obtaining the data information of the real-time input data input into the AI all-in-one machine system according to a sampling interval, generating a delay, and an average compression rate.

[0100]

[0101] ​​The real-time input data refers to data received by the AI all-in-one machine system in real time during running, and examples include real-time collected sensor data, user uploaded inference task data, and externally synchronized training sample data. The data information includes data frame information and data frame size in the real-time input data. The generation delay refers to the time interval from data generation to input into the AI all-in-one machine system. The average compression rate refers to the compression ratio of data after compression algorithm processing, i.e., the ratio of the size of compressed data to the size of original data.

[0102] In step 402, the correlation parameter of the real-time input data and the data acquired in the previous sampling interval is determined based on the data information of the real-time input data and the generation delay, and the storage priority of the real-time input data is determined based on the correlation parameter and the average compression rate.

[0103] The storage priority is used to determine the initial storage level of the real-time input data.

[0104] It should be noted that the real-time input data does not have access features in training and inference tasks, and therefore needs to be analyzed for its own data importance for storage level division. The input of general data is in units of data frames, and the higher the sensitivity of the data frame, the higher the upper layer to which it should be stored. When analyzing the importance of the data block of real-time data, the present application can first determine whether it is stream data or static data. Since stream data usually needs to be processed and analyzed in real time and the correlation between different data frames is relatively large, static data does not require real-time and has a relatively small correlation between data frames. That is, the more likely the data frame is stream data, the higher the correlation of its data, and therefore it is more suitable for storage through the upper layer.

[0105] For example, the correlation parameter can be represented by the following formula:

[0106]

[0107] is the correlation parameter of the data acquired at the current time and the previous sampling interval, is the maximum principal component vector of all data frame data in the sampling interval at the current time, which can be determined based on the data information of the real-time input data by principal component analysis. is the maximum principal component vector of all data frame data in the sampling interval at the previous time. is the average generation delay of the real-time input data acquired in the sampling interval at the current time, is the average generation delay of the real-time input data acquired in the sampling interval at the previous time, is an infinitesimal positive number. ​​​​

[0108] The above represents the content correlation of the data in two adjacent sampling intervals, and the greater the value is, the more similar the data content of different data frames in the stream data is to the static data, and the greater the corresponding correlation parameter is, represents the average generation delay of the data in two adjacent sampling intervals, and the generation delay between the stream data frames is generally consistent, that is The smaller the value is, the more likely the data between the frames is stream data, and the greater the correlation parameter is.

[0109] It should be noted that the stream data has different effects after being input into the AI all-in-one machine, or is used for model training, or is directly archived, and the data used for training needs more obvious access features than the archived data, which should be stored in a higher layer. Since the data usually needs to be compressed during transmission, and the model training needs data with smaller compression rate, and the archived data needs higher compression rate. Therefore, the application can evaluate the role of real-time input data based on the average compression rate of the real-time input data. And the storage priority of the real-time input data is determined by combining the correlation of the data frames.

[0110] For example, the storage priority can be represented by the following formula:

[0111]

[0112] Wherein, is the storage priority of the real-time input data in the sampling interval at the moment. is the correlation parameter of the data obtained in the sampling interval at the moment and the previous moment, is the average compression rate of the real-time input data in the sampling interval at the moment. That is, if the correlation parameter is greater, and the compression rate is smaller (the feature used for model training is more obvious), then the storage priority of the real-time input data is higher, and it should be stored in a higher layer.

[0113] Step 403, store the real-time input data to the corresponding storage level according to the storage priority.

[0114] In some embodiments, the value range of the correlation parameter is divided into multiple intervals, and the multiple intervals correspond to the storage levels in the AI all-in-one machine system one by one, and the initial storage level of the real-time input data is the storage level corresponding to the interval where the correlation parameter of the real-time input data is located.

[0115] ​​​For example, the storage priority of the real-time input data can be determined by the above method, the result is normalized to the range of [0, 1], the double threshold is determined to be 0.25 and 0.85 to divide into three intervals, which correspond to three storage levels respectively, and then the data is stored in the corresponding storage level based on the double threshold and indexed for subsequent access.

[0116] It should be noted that the above steps 401-403 are used for storage decision of real-time input data, and thus are suitable for application scenarios in which the AI all-in-one machine system receives input in real time. The above steps 101-103 are periodically executed based on the migration period, and thus the execution order of the above steps 401-403 and the above steps 101-103 is not limited.

[0117] Based on the above technical solution, the multi-dimensional feature parameters of the real-time input data are obtained in the embodiment of the application to provide a comprehensive basis for determining the initial storage level. The importance and real-time requirement of the real-time data are quantified by calculating the correlation parameter and the storage priority. Finally, the real-time input data is stored in the corresponding storage level based on the storage priority, so that the real-time input data can be initially stored in the adaptive storage level. The above solution solves the problem that the real-time input data cannot be determined in the storage level due to no access history, avoids the performance waste or deficiency caused by blind storage of new data, lays a good foundation for subsequent dynamic update based on the migration strategy, and further improves the overall data storage efficiency of the AI all-in-one machine system.

[0118] It should be noted that the above-mentioned embodiment sequence of the application is only for description, and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or can be advantageous.

[0119] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments.

Claims

1. A data storage method of an AI all-in-one system, characterized by, The method comprises: determining access and demand statistical characteristics of data blocks in different storage levels in the AI all-in-one system; the access and demand statistical characteristics comprise access frequency of a data block in the current migration period, time of the last access to the data block in the current migration period, and storage duration of the data block in the storage level; establishing a corresponding cross-level data migration strategy by analyzing the access and demand statistical characteristics of the data blocks in different storage levels; the cross-level data migration strategy is used to represent the target storage level of each data block in the next migration period; dynamically updating the storage level of each data block when the migration period is switched based on the cross-level data migration strategy; the cross-level data migration strategy is established by analyzing the access and demand statistical characteristics of the data blocks in different storage levels, comprising: for each data block, calculating a heat value of the data block at the end of the current migration period according to the access frequency of the data block, the time of the last access to the data block in the current migration period, and the storage duration of the data block in the storage level; based on the heat value of each data block and the heat average of each storage level, screening out an abnormal data block set in each storage level; for each abnormal data block set, determining the migration direction of the data blocks in the abnormal data block set, and calculating the migration matching degree of the data blocks in the abnormal data block set and each candidate storage level; the candidate storage level is the storage level in the migration direction of the corresponding data block; determining the target storage level of the data block in the next migration period according to the migration matching degree, so as to establish the cross-level data migration strategy.

2. The method of claim 1, wherein, based on the heat value of each data block and the heat average of each storage level, screening out an abnormal data block set in each storage level, comprising: determine the heat abnormality of each data block based on the heat value of each data block and the heat average of each storage level, and cluster according to the deviation between the heat abnormality of different data blocks to screen out an abnormal data block set in each storage level.

3. The method of claim 1, wherein, The migration direction is determined based on the heat value of the data block in the last migration period and the heat value in the current migration period; The migration matching degree of the data block and the candidate storage level is determined based on the heat value of the data block in the current migration period, the heat average of the candidate storage level, and the data size of the data in the candidate storage level belonging to the same data file as the data block.

4. The method of claim 1, wherein, In the case where there is no storage level in the migration direction of the data block, the target storage level of the data block in the next migration period is the storage level of the data block in the current migration period.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: acquiring data information of real-time input data input into the AI all-in-one system in real time, generating delay and average compression rate according to a sampling interval; The data information based on the real-time input data and a delay are used to determine a correlation parameter between the real-time input data and data obtained in a previous sampling interval, and the storage priority of the real-time input data is determined based on the correlation parameter and the average compression rate; the storage priority is used to determine an initial storage level of the real-time input data; The real-time input data is stored in a corresponding storage level according to the storage priority.

6. The method of claim 5, wherein, The value range of the storage priority is divided into multiple intervals; the multiple intervals correspond to the storage levels in the AI all-in-one system one by one, and the initial storage level of the real-time input data is the storage level corresponding to the interval where the storage priority of the real-time input data is located.

7. The method according to any one of claims 1 to 4, characterized in that, The storage levels in the AI all-in-one system are divided in order from high to low according to data reading performance.

8. The method of claim 7, wherein, The storage levels in the AI all-in-one system include a hot data layer, a warm data layer, and a cold data layer; the data reading performance of the hot data layer is greater than that of the warm data layer; and the data reading performance of the warm data layer is greater than that of the cold data layer.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, and when a computer executes the instructions, the computer executes the data storage method of the AI all-in-one system according to any one of claims 1-8.

Images