Data storage method based on data quality, medium and equipment

Abstract

The invention discloses a data storage method based on data quality, a medium and equipment. An initial data set is obtained. And for each piece of data in the initial data set, inputting the data into each preset labeling model to obtain a labeling result of each labeling model for the data, and taking the labeling result as a labeling set corresponding to the data. And according to the label set corresponding to each piece of data in the initial data set, performing data screening in the initial data set to construct a target data set. Determining data access information of a preset computing node when executing a model training task of the target model, and determining data characteristic information corresponding to each piece of data in the target data set, according to the data access information and the data characteristic information corresponding to each piece of data contained in the target data set; and determining a storage position of each data in the target data set when the computing node executes the model training task of the target model, and storing the target data set as a training sample according to the storage position.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a data storage method, medium, and device based on data quality. Background Art

[0002] At present, with the rapid development of artificial intelligence, the large models trained in this way have achieved great success in fields such as computer vision, natural language processing or recommendation systems, such as semantic understanding and generation in natural language processing, image recognition and classification in computer vision, etc.

[0003] However, this tremendous success also comes with a problem: training large models requires a vast amount of training data and computing resources. After acquiring training data, it is typically stored on the compute nodes performing model training tasks, as well as in off-node memory, allowing for easy access during model training. However, the compute nodes' inefficiency in accessing training data reduces model training efficiency.

[0004] To this end, this specification provides a data storage method, medium, and device based on data quality. Summary of the Invention

[0005] This specification provides a data storage method, medium and device based on data quality to partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions: This specification provides a data storage method based on data quality, including: Get the initial dataset; For each data in the initial data set, the data is input into each preset annotation model respectively, and the annotation result of each annotation model for the data is obtained as the annotation set corresponding to the data; According to the annotation set corresponding to each data in the initial data set, data is screened in the initial data set to construct a target data set; Determine data access information for a preset computing node when executing a model training task for a target model, and determine data characteristic information corresponding to each data in the target data set, wherein the data access information is used to reflect the usage of various types of training samples by the computing node when executing the model training task for the target model. For each data contained in the target data set, the data characteristic information corresponding to the data is used to reflect the type of the data and the association relationship between the data and other data; Based on the data access information and the data characteristic information corresponding to each data contained in the target data set, the storage location of each data in the target data set is determined when the computing node executes the model training task of the target model, and the target data set is stored as a training sample according to the storage location.

[0007] Optionally, for each data in the initial data set, the data is input into each preset annotation model, specifically including: Determining data characteristic information of each data in the initial data set; Determining, based on the data characteristic information, a classifier that matches the initial data set, the classifier being used to distinguish data quality of each data in the initial data set; Input each data in the initial data set into the classifier respectively to obtain a classification result of the data quality corresponding to each data in the initial data set; According to the classification result, data with higher than a preset data quality is screened out from the initial data set to construct an intermediate data set; For each data in the intermediate data set, the data is input into each preset annotation model respectively.

[0008] Optionally, based on the annotation set corresponding to each data in the initial data set, data screening is performed in the initial data set to construct a target data set, specifically including: For each data in the initial data set, according to the annotation set corresponding to the data, the data is combined with each annotation result in the annotation set to obtain each filtered data; Inputting each of the screening data into a preset model respectively, and determining each feedback information output by the preset model; Determining, from among the screening data, the screening data whose feedback information meets preset expectations based on the feedback information; A target data set is constructed based on the filtered data that meets the preset expectations of the feedback information.

[0009] Optionally, determining, based on the data access information and data characteristic information corresponding to each data contained in the target data set, a storage location of each data in the target data set when the computing node executes the model training task of the target model, and storing the target data set as a training sample according to the storage location, specifically includes: Determining, based on the data access information, the type of training data required for each round of model training when the computing node performs the model training task of the target model, and the order in which the training data is acquired; Determining, from the target data set, the training samples required for each round of model training when the computing node performs the model training task of the target model, based on the type of training data required for each round of training and data characteristic information; Determine the storage location of data other than the determined training samples in the target data set in the storage node and store them, and determine the storage location of the training samples required for each round of model training in the cache layer of the computing node and store them according to the acquisition order, wherein the rate at which the computing node obtains data from the cache layer is greater than the rate at which the computing node obtains data from the storage node.

[0010] Optionally, determining, based on the data access information and data characteristic information corresponding to each data contained in the target data set, a storage location of each data in the target data set when the computing node executes the model training task of the target model, and storing the target data set as a training sample according to the storage location, specifically includes: Determining, based on the data access information, the type of training data required for each round of model training when the computing node performs the model training task of the target model, and the order in which the training data is acquired; Determining, from the initial data set, the intermediate data set, and the target data set, the training samples required for each round of model training when the computing node performs the model training task of the target model, according to the type of training data required for each round of training and data characteristic information; Storing the determined training samples in the first cache layer of the computing node in sequence according to the acquisition order; Determining, from the initial data set, data other than the determined training sample as a first remaining training sample, determining, from the intermediate data set, data other than the determined training sample as a second remaining training sample, and determining, from the target data set, data other than the determined training sample as a third remaining training sample; According to the association relationship contained in the data characteristic information, the data in the third remaining training samples are stored in the first cache layer of the computing node, the data in the second remaining training samples are stored in the second cache layer of the computing node, and the data in the first remaining training samples are stored in the storage node, wherein the rates at which the computing node obtains data are, from fast to slow, the rate at which data is obtained from the first cache layer, the rate at which data is obtained from the second cache layer, and the rate at which data is obtained from the storage node.

[0011] Optionally, the method further includes: Obtaining intermediate results generated by each round of model training when the computing node executes the model training task of the target model; For the intermediate results generated by each round of model training, determine whether the intermediate results of this round of model training are called within the preset time based on the calling regularity information of the intermediate results of this round of model training; If yes, storing the intermediate results of the round of model training in the first cache layer of the computing node, so that the computing node can access the intermediate results of the round of model training from the first cache layer; If not, the intermediate results of this round of model training are compressed and stored in the second cache layer of the computing node, so that the computing node can access the intermediate results of this round of model training from the second cache layer.

[0012] Optionally, the method further includes: When the number of model training rounds of the model training task of the target model executed by the computing node reaches a preset number of rounds, the target model is trained using the third remaining training samples stored in the first cache layer.

[0013] Optionally, determining, based on the data access information and data characteristic information corresponding to each data contained in the target data set, a storage location of each data in the target data set when the computing node executes the model training task of the target model, specifically includes: determining, based on the data access information, an initial storage location of each data in the target data set; determining, based on data characteristic information corresponding to each data item in the target data set, a call efficiency value when each data item in the target data set is stored according to the initial storage location; With maximizing the call efficiency value as the optimization goal, the initial storage location of each data in the target data set is adjusted to determine the storage location of each data in the target data set when the computing node executes the model training task of the target model.

[0014] This specification provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned data storage method based on data quality.

[0015] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a data storage method based on data quality when executing the program.

[0016] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects: The data storage method based on data quality provided in this specification obtains an initial data set. For each data in the initial data set, the data is input into each preset annotation model respectively, and the annotation result of each annotation model for the data is obtained as the annotation set corresponding to the data. According to the annotation set corresponding to each data in the initial data set, data is screened in the initial data set to construct a target data set. The data access information of the preset computing node when executing the model training task of the target model is determined, and the data characteristic information corresponding to each data in the target data set is determined. The data access information is used to reflect the usage of various types of training samples by the computing node when executing the model training task for the target model. For each data contained in the target data set, the data characteristic information corresponding to the data is used to reflect the type of the data and the association relationship between the data and other data. According to the data access information and the data characteristic information corresponding to each data contained in the target data set, the storage location of each data in the target data set is determined when the computing node executes the model training task of the target model, and the target data set is stored as a training sample according to the storage location.

[0017] By labeling the acquired initial data set, the data quality of the training samples when the computing node performs the model training task is improved. In addition, by using the data access information of the target model during the computing node's execution of the model training task and the data characteristic information of the data required for training, the storage location of each data in the target data set suitable for the model training task is determined. This allows the computing node to quickly obtain training samples from a pre-determined and reasonable storage location when executing the model training task of the target model, thereby increasing the efficiency of data supply and model training and improving the training quality of the target model. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings: Figure 1 A flowchart of a data storage method based on data quality provided in an embodiment of this specification; Figure 2 A schematic diagram of a process for constructing a target dataset provided in this specification; Figure 3 A schematic diagram of a storage framework provided in this specification; Figure 4 A schematic diagram of a data storage device based on data quality provided in this specification; Figure 5 This manual provides a corresponding Figure 1 Schematic diagram of the structure of the electronic equipment. DETAILED DESCRIPTION

[0019] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0020] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0021] Figure 1 A flowchart of a training sample storage method provided in an embodiment of this specification includes the following steps: S100: Obtain an initial data set.

[0022] In this specification, the process of data labeling and training sample storage is performed. In the embodiments of this specification, this training sample storage process can be performed by a server. Of course, this specification does not limit the device used to perform the training sample storage process. Devices such as personal computers and mobile terminals can also be used to perform data labeling and storage. For ease of description, the following description uses the server as the execution entity.

[0023] In one or more embodiments of this specification, since training a large model requires a large amount of training data, the server can obtain a large amount of data by crawling the required data through crawler technology or obtaining data through relevant databases, and use this data to construct an initial data set in the specification.

[0024] S102: For each data in the initial data set, the data is input into each preset annotation model respectively to obtain the annotation result of each annotation model for the data as the annotation set corresponding to the data.

[0025] In one or more embodiments of this specification, after obtaining an initial dataset, the server attempts to improve the data quality of the initial dataset in order to make it more suitable for model training tasks. The most important factor affecting data quality is data labeling, and the quality of data labeling directly determines the upper limit of the effectiveness of data use. Therefore, for each data point in the initial dataset, the data point can be input into each preset labeling model, and the labeling results of each labeling model for the data point can be obtained, which serve as the labeling set corresponding to the data point.

[0026] In one or more embodiments of the present specification, multiple preset labeling models can be used to label the same data separately. In order to increase the accuracy of the labeling results, these multiple labeling models may not be the same labeling model, such as labeling models based on decision trees, support vector machines, random forests, and neural networks.

[0027] In one or more embodiments of this specification, the initial data set obtained by the server contains a large amount of data, and these data may have various problems. If used directly to train the model, it may lead to low model training efficiency and low performance of the trained model, and therefore cannot be used directly to train the model. Problems that may exist with these data include the following: some data are missing key parts, and the missing values ​​need to be filled, while some parts of the data are abnormal and the abnormal values ​​need to be processed. Therefore, before the initial data set is input into the preset annotation models, the data in the initial data set can be preprocessed. In this specification, there is no restriction on the means of preprocessing, which can be set according to actual conditions.

[0028] Furthermore, to better utilize the data to train the target model, it is also possible to understand the data characteristics of the data for targeted utilization. In this specification, Exploratory Data Analysis (EDA) techniques can be used to analyze and process the data in the initial dataset to obtain data characteristic information of the data in the initial dataset. Data characteristic information can include explicit characteristic information (such as distribution, outliers, and types, etc.) and implicit characteristic information.

[0029] Explicit characteristic information refers to the frequency of words in a piece of data. Type refers to the type of data determined based on word frequency. An outlier indicates an abnormality in a piece of data, such as a data format or size. Implicit characteristic information refers to potential connections between data, such as whether data is similar and determining which data are related.

[0030] In one or more embodiments of the present specification, the server processes a relatively large amount of data in the acquired initial data set through various preset annotation models. The processing time increases with the increase in the amount of data, but the annotation efficiency of the annotation model does not increase with the increase in the amount of data. Therefore, before this, after determining the data characteristic information of each data in the initial data set, the server determines a classifier that matches the initial data set based on the data characteristic information. Each data in the initial data set is then input into the classifier to obtain a classification result of the data quality corresponding to each data in the initial data set output by the classifier. Based on the classification result, data with a quality higher than the preset data quality is filtered out from the initial data set to construct an intermediate data set.

[0031] The classifier is used to distinguish the data quality of each item in the initial dataset. Then, data with classification results exceeding the preset data quality are filtered out from the initial dataset to determine the intermediate dataset. Each item in the intermediate dataset is then fed into the preset annotation models to determine the target dataset.

[0032] Figure 2 This is a flow chart of constructing a target dataset provided in this manual. Figure 2 As shown, the server preprocesses the initial dataset, determines a classifier that matches the preprocessed dataset, and uses this classifier for classification. Based on the classification results output by the classifier, the server determines an intermediate dataset, inputs the intermediate dataset into each preset annotation model, and determines the final annotation result from the annotation results of each annotation model, thereby constructing the target dataset.

[0033] S104: According to the annotation set corresponding to each data in the initial data set, data is screened in the initial data set to construct a target data set.

[0034] In one or more embodiments of the present specification, the server filters the initial data set according to the annotation set corresponding to each data in the initial data set to construct a target data set.

[0035] Specifically, the server can use an integrated detection mechanism, such as a voting mechanism, to identify the most frequently appearing annotation result among the annotation results of each annotation model for the same data item in the annotation set. This result is then used as the final annotation result for the data item. The data with the final annotation result is then filtered out from the initial dataset to form the target dataset.

[0036] Of course, in this specification, the server may also combine each data point in the initial dataset with each annotation result in the annotation set according to the annotation set corresponding to the data point to obtain each filtered data point. Each filtered data point is input into a preset model to determine each feedback information output by the preset model. Based on each feedback information, the server determines the filtered data point whose feedback information meets the preset expectations from the filtered data point, and then constructs the target dataset based on the filtered data point whose feedback information meets the preset expectations.

[0037] Among them, the feedback information can be information such as training time or model loss value, which is not limited in this specification. The feedback information is used to reflect the specific performance of the model trained using the filtered data. On this basis, the server can use the labeling results in the filtered data that meet the preset expectations, that is, the labeling results that determine the specific performance of the training model with better results, as the final labeling results of the data.

[0038] In one or more embodiments of the present specification, after determining the labeling results in the filtered data that meet the preset expectations, the server can train the labeling model whose output feedback information does not meet the preset expectations based on the labeling results in the filtered data that meet the preset expectations, and optimize the labeling model so that the labeling results output by the labeling model are more accurate.

[0039] Specifically, the server may determine, from the various filtered data, filtered data whose feedback information does not meet preset expectations based on the various pieces of feedback information. Based on the filtered data whose feedback information does not meet preset expectations, the server may determine a labeling model that outputs labeling results whose feedback information does not meet preset expectations, as the model to be trained. Furthermore, the server may determine a loss value based on the difference between the labeling results in the filtered data whose feedback information meets preset expectations and the labeling results output by the model to be trained, and train the model to be trained based on the loss value, such as minimizing the loss value, as an optimization goal, wherein the difference is positively correlated with the loss value.

[0040] S106: Determine the data access information of the preset computing node when executing the model training task of the target model, and determine the data characteristic information corresponding to each data in the target data set. The data access information is used to reflect the usage of various types of training samples by the computing node when executing the model training task for the target model. For each data included in the target data set, the data characteristic information corresponding to the data is used to reflect the type of the data and the association relationship between the data and other data.

[0041] In one or more embodiments of the present specification, after determining the target data set with annotations, the server, in order to better utilize the target data set and give full play to the role of the target data set, enables the computing node to have higher training efficiency when using the target data set to perform the model training task of the target model. The server can determine the data access information of the preset computing node when performing the model training task of the target model. Since the computing node itself is clear about the structure and parameters of the target model and the usage of various training samples by the computing node when performing the model training task for the target model when determining to train the target model, when each round of training of the target model is performed, the server can determine in advance the training data type required for the next round of training and the order of obtaining the training data, i.e., data access information, based on the usage of various training samples by the computing node when performing the model training task for the target model. In addition, the server can also determine, for each data contained in the target data set, data characteristic information reflecting the type of the data and the relationship between the data and other data, i.e., determine the data characteristic information corresponding to each data in the target data set. It is worth noting that the various training samples at this time refer to training samples of various data types.

[0042] Therefore, after determining the data access information and the data characteristic information, the server can store the data of the target data set according to the data access information and the data characteristic information so as to facilitate the computing node call.

[0043] S108: Based on the data access information and the data characteristic information corresponding to each data contained in the target data set, determine the storage location of each data in the target data set when the computing node executes the model training task of the target model, and store the target data set as a training sample according to the storage location.

[0044] In one or more embodiments of the present specification, the server determines the storage location of each data in the target data set when the computing node executes the model training task of the target model according to the determined data access information and the data characteristic information corresponding to each data contained in the target data set, and stores the target data set as a training sample according to the storage location.

[0045] Specifically, based on the data access information, the server determines the type of training data required for each round of model training when the computing node executes the target model training task, as well as the order in which the training data is acquired. Based on the type of training data required for each round of training and the data characteristics information, the server determines, from the target dataset, the training samples required for each round of model training when the computing node executes the target model training task. The server can then determine the storage location of data in the target dataset other than the determined training samples in the storage node and store them. Furthermore, the server can determine the storage location of the training samples required for each round of model training in the cache layer of the computing node according to the acquisition order and store them.

[0046] The rate at which the computing node obtains data from the cache layer is greater than the rate at which the computing node obtains data from the storage node. Of course, when storing data other than the determined training samples in the target dataset in the storage node, the data with related relationships can be stored together based on the association relationships contained in the data characteristic information.

[0047] In this case, the storage location determined by the server can be encompassed by a pre-defined storage framework. This framework consists of two parts: a cache layer within the compute nodes and storage nodes. The cache layer within the compute nodes can be composed of high-performance storage devices such as dynamic random access memory (DRAM) and high-bandwidth memory (HBM), as well as non-volatile storage devices such as non-volatile memory express (NVMe) solid-state drives. Storage nodes can be parallel file systems comprised of conventional large-capacity storage devices such as Serial Advanced Technology Attachment (SATA) / Serial Attached SCSI (SAS) solid-state drives and hard disk drives (HDDs). SCSI stands for Small Computer System Interface. Within the storage framework, the cache layer within the upper-tier compute nodes acts as a high-speed cache for the lower-tier storage nodes. Together, they maintain a unified directory tree view through a metadata server, establishing a unified addressing space across different storage tiers. The entire storage framework belongs to the same namespace (i.e., a unified addressing space), aggregates and stores data or files, and rationally utilizes the characteristics of high-performance storage media to accelerate the reading and writing of small files. The same data usually only needs to be stored once, maximizing data sharing and storage space utilization. The upper cache layer can be built based on new high-speed interconnection technologies. In addition, the construction of the storage framework can reuse existing storage technologies, such as fine-grained distributed lock management and version control, to achieve a balance between data security, system performance, and cost. This manual does not limit storage technologies.

[0048] In addition, in one or more embodiments of the present specification, the server may also determine the type of training data required for each round of model training when the computing node performs the model training task of the target model, and the order in which the training data is acquired based on the data access information. The server determines the training samples required for each round of model training when the computing node performs the model training task of the target model from the initial data set, the intermediate data set, and the target data set based on the type of training data required for each round of training and the data characteristic information. Afterwards, the server stores the determined training samples in the first cache layer of the computing node in the order of acquisition, determines the data other than the determined training samples from the initial data set as the first remaining training samples, determines the data other than the determined training samples from the intermediate data set as the second remaining training samples, and determines the data other than the determined training samples from the target data set as the third remaining training samples.

[0049] Afterwards, the server can store the data in the third remaining training samples to the first cache layer of the computing node, the data in the second remaining training samples to the second cache layer of the computing node, and the data in the first remaining training samples to the storage node based on the association relationship contained in the data characteristic information, wherein the rate at which the computing node obtains data is, from fast to slow, the rate at which data is obtained from the first cache layer, the rate at which data is obtained from the second cache layer, and the rate at which data is obtained from the storage node.

[0050] In this case, the storage framework consists of three layers: the first and second cache layers of the compute nodes, and the storage nodes. The first cache layer located on the compute nodes can be composed of high-performance storage devices such as DRAM and HBM, while the second cache layer located on the compute nodes can be composed of non-volatile storage devices such as NVMe solid-state drives.

[0051] In one or more embodiments of the present specification, further, the server may obtain the intermediate results generated by each round of model training when the computing node executes the model training task of the target model, temporary data, status information, and activation values ​​generated during the model training process. For the intermediate results generated by each round of model training, based on the calling rule information of the intermediate results of the round of model training in the round of model training, it is determined whether the intermediate results of the round of model training are called within a preset time. Among them, the calling rule information refers to the pattern displayed by the subsequent access and use of the intermediate results generated by each round of model training. The calling rule information can record information such as the time interval from the generation of the intermediate results of a certain round of model training to the next call of the intermediate results.

[0052] If so, it indicates that this intermediate result will be called multiple times during this round of model training. The intermediate result of this round of model training should be stored in the first cache layer of the computing node, so that the computing node can directly access the intermediate result of this round of model training from the first cache layer when it needs to call the intermediate result. After the computing node accesses the intermediate result, the intermediate result in the first cache layer is discarded, but the intermediate result can be stored in the storage node. If not, it indicates that this intermediate result will not be called multiple times during this round of model training. After this intermediate result is generated, it may take some time before it is called. In order to save storage space for storing this intermediate result, the intermediate result of this round of model training can be compressed and stored in the second cache layer of the computing node, so that the computing node can directly access the intermediate result of this round of model training from the second cache layer or storage node when it needs to call the intermediate result. After this intermediate result is stored in the second cache layer and accessed, it is discarded from the second cache layer, but this intermediate result needs to be stored in the storage node. Of course, the computing node can access data from the first cache layer, the second cache layer, and any storage location in the storage node based on the Storage Performance Development Kit (SPDK) technology, and this specification does not limit the access technology of the computing node directly accessing data at any storage location.

[0053] It is worth noting that, in this specification, before a compute node executes a target model training task, the data is stored in the corresponding storage location based on data access information and data characteristics. The server then uses this stored data to enable the compute node to execute the target model training task. However, in the later stages of the target model training task (when the number of model training rounds reaches a certain number or the model accuracy meets certain requirements), high-quality data (data from the target dataset stored in the first cache layer) is supplied to the compute node as much as possible to accelerate model training.

[0054] Specifically, when the number of model training rounds of the computing node executing the model training task of the target model reaches a preset number of rounds, the server can enable the computing node to train the target model through the third remaining training samples stored in the first cache layer.

[0055] In one or more embodiments of the present specification, when data in a target data set is stored according to data characteristic information, the storage locations of related data in the target data set may be adjacent. When a computing node calls the stored data of the target data set as a training sample during the execution of a model training task, if the data is read in sequence according to the storage locations of the stored training samples, the training samples read are not necessarily the training samples that maximize the model training efficiency.

[0056] To this end, the server can determine the initial storage location of each data item in the target dataset based on the data access information, and determine the call efficiency value when storing each data item in the target dataset at the initial storage location based on the data characteristic information corresponding to each data item in the target dataset. The call efficiency value is used to reflect the model training efficiency when the computing node calls the training sample at the initial storage location. The higher the call efficiency value, the less time the computing node needs to call the training sample at the initial storage location, and the higher the model training efficiency.

[0057] The server can then optimize the target by maximizing the call efficiency value and adjust the initial storage location of each data point in the target dataset. This determines the storage location for each data point when the compute node executes the target model's model training task. This process can be understood as a continuous search for an optimal solution (the so-called optimal solution is the optimal storage location for each training sample while maximizing model training efficiency). By setting the constraint of maximizing the call efficiency value, various possible storage locations for each data point can be continuously explored, ultimately determining the storage location that maximizes model training efficiency and storing each data point in the target dataset accordingly.

[0058] In addition, in one or more embodiments of this specification, the storage framework not only stores the data required for model training, but also stores the model parameters of the model run by the computing node, and also stores the intermediate results generated during the model operation. Through this storage framework, the computing node can directly access data from the first cache layer and the second cache layer. The first cache layer can store compressed model parameters (model parameters that are being used or will be used), and the second cache layer can store uncompressed model parameters (parameters that are not needed for a period of time). It can achieve a balance between data volume, latency and throughput requirements, and the model parameters and data required for training only need to be stored in a certain cache layer, eliminating the problems of storage space waste caused by multi-layer storage in the traditional hierarchical cache architecture, the need for inter-layer migration and synchronization overhead, and comprehensively improving the cache storage space utilization and accelerating the efficiency of large model training.

[0059] It is worth noting that during the model training process of the computing node, specific precision is specified for some special functions and model structures involved in the model training (for example, when calculating gradients, higher precision is used to ensure the accuracy of model training. Or when calculating matrix multiplication, lower precision is used to significantly reduce computing power requirements and accelerate calculations), so as to adapt to the data range requirements of special functions. For unspecified general functions and model structures, an online learning neural network can be used to guide the variable precision that should be used in the next round of operations, thereby achieving high-accuracy data precision adaptive adjustment and reducing the storage space occupied by intermediate results generated during the model training process.

[0060] Figure 3 This is a schematic diagram of a storage framework provided in this manual. Figure 3 As shown, the storage framework includes a first cache layer and a second cache layer located in the compute nodes, as well as a massively parallel file system located in the storage nodes. The compute nodes contain several compute cores, which perform model training. During training, they can directly access the first cache layer for high-speed reading and writing of training data, enabling efficient data exchange. The compute cores can also access the second cache layer across layers to obtain persistent, large-capacity training data stored there. They can also adaptively compress and store model parameters generated during model training in the second cache layer, pre-preparing the training data and model parameters that the compute cores will need. Data can also be migrated between the first and second cache layers based on data acquisition needs. The massively parallel file system located in the storage node has its own massive storage space, which aggregates bandwidth to accelerate data access efficiency and supports parallel reading and writing of data, resulting in high throughput. It can also interconnect with the first and second cache layers at high speed, sending its own stored data to the compute nodes and receiving data sent by the compute nodes for updating or storage.

[0061] The above is a data storage method based on data quality provided by one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding data storage device based on data quality, such as Figure 4 shown.

[0062] Figure 4 A schematic diagram of a data storage device based on data quality provided in this specification, specifically including: An acquisition module 400 is used to acquire an initial data set; The annotation module 402 is configured to input each data in the initial data set into each preset annotation model, and obtain the annotation result of each annotation model for the data as the annotation set corresponding to the data; A screening module 404 is configured to screen data in the initial data set according to the annotation set corresponding to each data in the initial data set to construct a target data set; Determination module 406 is used to determine data access information of a preset computing node when executing a model training task for a target model, and determine data characteristic information corresponding to each data in the target data set. The data access information is used to reflect the usage of various training samples by the computing node when executing the model training task for the target model. For each data included in the target data set, the data characteristic information corresponding to the data is used to reflect the type of the data and the relationship between the data and other data. The storage module 408 is used to determine the storage location of each data in the target data set when the computing node executes the model training task of the target model based on the data access information and the data characteristic information corresponding to each data contained in the target data set, and store the target data set as a training sample according to the storage location.

[0063] Optionally, the labeling module 402 is used to determine data characteristic information of each data in the initial data set, and determine a classifier that matches the initial data set based on the data characteristic information. The classifier is used to distinguish the data quality of each data in the initial data set, and each data in the initial data set is input into the classifier respectively to obtain a classification result of the data quality corresponding to each data in the initial data set. Based on the classification result, data with a higher than preset data quality is screened out from the initial data set to construct an intermediate data set, and for each data in the intermediate data set, the data is input into each preset labeling model respectively.

[0064] Optionally, the screening module 404 is used to combine each data in the initial data set with each annotation result in the annotation set according to the annotation set corresponding to the data to obtain each screening data, input each screening data into the preset model respectively, determine each feedback information output by the preset model, and determine the screening data whose feedback information meets the preset expectations from the each screening data according to the feedback information, and construct the target data set based on the screening data whose feedback information meets the preset expectations.

[0065] Optionally, the storage module 408 is used to determine, based on the data access information, the type of training data required for each round of model training when the computing node performs the model training task of the target model, and the order in which the training data is acquired; determine, from the target data set, the training samples required for each round of model training when the computing node performs the model training task of the target model, based on the type of training data required for each round of training and the data characteristic information; determine and store the storage location of the data in the target data set other than the determined training samples in the storage node; determine and store the storage location of the training samples required for each round of model training in the cache layer of the computing node according to the acquisition order; wherein, the rate at which the computing node acquires data from the cache layer is greater than the rate at which the computing node acquires data from the storage node.

[0066] Optionally, the labeling module 402 is used to determine, based on the data access information, the type of training data required for each round of model training when the computing node performs the model training task of the target model, and the order in which the training data is obtained; based on the type of training data required for each round of training and the data characteristic information, determine, from the initial data set, the intermediate data set and the target data set, the training samples required for each round of model training when the computing node performs the model training task of the target model; store the determined training samples in the first cache layer of the computing node in sequence according to the acquisition order; determine, from the initial data set, data other than the determined training samples as the first remaining training samples; In the intermediate data set, data other than the determined training samples are determined as second remaining training samples, and from the target data set, data other than the determined training samples are determined as third remaining training samples. According to the association relationship contained in the data characteristic information, the data in the third remaining training samples are stored in the first cache layer of the computing node, the data in the second remaining training samples are stored in the second cache layer of the computing node, and the data in the first remaining training samples are stored in the storage node, wherein the rates at which the computing node obtains data are, from fast to slow, the rate at which data is obtained from the first cache layer, the rate at which data is obtained from the second cache layer, and the rate at which data is obtained from the storage node.

[0067] Optionally, the labeling module 402 is used to obtain the intermediate results generated by each round of model training when the computing node performs the model training task of the target model. For the intermediate results generated by each round of model training, based on the calling rule information of the intermediate results of the round of model training in the round of model training, it is determined whether the intermediate results of the round of model training are called within a preset time. If so, the intermediate results of the round of model training are stored in the first cache layer of the computing node, so that the computing node accesses the intermediate results of the round of model training from the first cache layer. If not, the intermediate results of the round of model training are compressed and stored in the second cache layer of the computing node, so that the computing node accesses the intermediate results of the round of model training from the second cache layer.

[0068] Optionally, the labeling module 402 is used to train the target model using the third remaining training samples stored in the first cache layer when the number of model training rounds of the model training task of the target model executed by the computing node reaches a preset number of rounds.

[0069] Optionally, the storage module 408 is used to determine the initial storage location of each data in the target data set based on the data access information, and determine the call efficiency value when each data contained in the target data set is stored according to the initial storage location based on the data characteristic information corresponding to each data contained in the target data set, so as to maximize the call efficiency value as the optimization goal, adjust the initial storage location of each data in the target data set, and determine the storage location of each data in the target data set when the computing node executes the model training task of the target model.

[0070] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides a data storage method based on data quality.

[0071] This manual also provides Figure 5 The schematic structure diagram of the electronic device shown in FIG. Figure 5 As shown, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The data storage method based on data quality.

[0072] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0073] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using physical hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly performed using software called a "logic compiler." This is similar to the software compilers used during program development. Before compilation, the original code must be written in a specific programming language, called a Hardware Description Language (HDL). There are many types of HDL, including ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that simply by programming a method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0074] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the memory control logic. Those skilled in the art will also appreciate that, in addition to implementing the controller purely in computer-readable program code, the controller can also be implemented in the form of logic gates, switches, an application-specific integrated circuit, a programmable logic controller, an embedded microcontroller, etc. by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the means for implementing the various functions included therein can also be considered as structures within the hardware component. Alternatively, the means for implementing the various functions can be considered both a software module implementing the method and a structure within the hardware component.

[0075] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0076] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0077] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0078] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0079] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0080] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0081] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0082] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0083] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0084] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0085] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0087] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0088] The foregoing is merely an example of the present invention and is not intended to limit the present invention. Various modifications and variations are possible within the scope of the present invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims of the present invention.

Claims

1. A data storage method based on data quality, characterized in that: include: Get the initial data set; For each data in the initial data set, the data is input into each preset annotation model respectively, and the annotation result of each annotation model for the data is obtained as the annotation set corresponding to the data; According to the annotation set corresponding to each data in the initial data set, data is screened in the initial data set to construct a target data set; Determine data access information of a preset computing node when executing a model training task of a target model, and determine data characteristic information corresponding to each data in the target data set, wherein the data access information is used to reflect the usage of various types of training samples by the computing node when executing a model training task for the target model, and for each data included in the target data set, the data characteristic information corresponding to the data is used to reflect the type of the data and the association relationship between the data and other data; Based on the data access information and the data characteristic information corresponding to each data contained in the target data set, the storage location of each data in the target data set is determined when the computing node executes the model training task of the target model, and the target data set is stored as a training sample according to the storage location.

2. The method according to claim 1, characterized in that For each data in the initial data set, the data is input into each preset annotation model, specifically including: Determining data characteristic information of each data in the initial data set; Determining, according to the data characteristic information, a classifier that matches the initial data set, the classifier being used to distinguish data quality of each data in the initial data set; Input each data in the initial data set into the classifier respectively to obtain a classification result of the data quality corresponding to each data in the initial data set; According to the classification result, data with higher than a preset data quality is screened out from the initial data set to construct an intermediate data set; For each data in the intermediate data set, the data is input into each preset annotation model respectively.

3. The method according to claim 1, characterized in that According to the annotation set corresponding to each data in the initial data set, data is screened in the initial data set to construct a target data set, specifically including: For each data in the initial data set, according to the annotation set corresponding to the data, the data is combined with each annotation result in the annotation set to obtain each screening data; Input each of the screening data into a preset model respectively, and determine each feedback information output by the preset model; According to the feedback information, from the screening data, determine the screening data whose feedback information meets the preset expectations; Construct a target data set based on the filtered data that meets the preset expectations of the feedback information.

4. The method according to claim 1, characterized in that Determining, according to the data access information and the data characteristic information corresponding to each data contained in the target data set, the storage location of each data in the target data set when the computing node executes the model training task of the target model, and storing the target data set as a training sample according to the storage location, specifically includes: Determine, according to the data access information, the type of training data required for each round of model training when the computing node performs the model training task of the target model, and the order in which the training data is acquired; Determine, from the target data set, the training samples required for each round of model training when the computing node performs the model training task of the target model according to the type of training data required for each round of training and data characteristic information; Determine the storage location of the data in the target data set other than the determined training samples in the storage node and store them, and determine the storage location of the training samples required for each round of model training in the cache layer of the computing node and store them according to the acquisition order, wherein the rate at which the computing node acquires data from the cache layer is greater than the rate at which the computing node acquires data from the storage node.

5. The method according to claim 2, characterized in that Determining, according to the data access information and the data characteristic information corresponding to each data contained in the target data set, the storage location of each data in the target data set when the computing node executes the model training task of the target model, and storing the target data set as a training sample according to the storage location, specifically includes: Determine, according to the data access information, the type of training data required for each round of model training when the computing node performs the model training task of the target model, and the order in which the training data is acquired; Determine, from the initial data set, the intermediate data set, and the target data set, the training samples required for each round of model training when the computing node performs the model training task of the target model according to the type of training data required for each round of training and data characteristic information; According to the acquisition order, the determined training samples are sequentially stored in the first cache layer of the computing node; Determine, from the initial data set, data other than the determined training sample as a first remaining training sample, determine, from the intermediate data set, data other than the determined training sample as a second remaining training sample, and determine, from the target data set, data other than the determined training sample as a third remaining training sample; According to the association relationship contained in the data characteristic information, the data in the third remaining training samples are stored in the first cache layer of the computing node, the data in the second remaining training samples are stored in the second cache layer of the computing node, and the data in the first remaining training samples are stored in the storage node, wherein the rates at which the computing node obtains data are, from fast to slow, the rate at which data is obtained from the first cache layer, the rate at which data is obtained from the second cache layer, and the rate at which data is obtained from the storage node.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining intermediate results generated by each round of model training when the computing node executes the model training task of the target model; For the intermediate results generated by each round of model training, determine whether the intermediate results of this round of model training are called within a preset time according to the calling rule information of the intermediate results of this round of model training in this round of model training; If yes, the intermediate result of the round of model training is stored in the first cache layer of the computing node, so that the computing node can access the intermediate result of the round of model training from the first cache layer; If not, the intermediate results of this round of model training are compressed and stored in the second cache layer of the computing node, so that the computing node can access the intermediate results of this round of model training from the second cache layer.

7. The method according to claim 5, characterized in that The method further comprises: When the number of model training rounds of the model training task of the target model executed by the computing node reaches a preset number of rounds, the target model is trained by using the third remaining training samples stored in the first cache layer.

8. The method according to claim 1, characterized in that Determining, according to the data access information and the data characteristic information corresponding to each data contained in the target data set, the storage location of each data in the target data set when the computing node executes the model training task of the target model, specifically includes: Determining, according to the data access information, an initial storage location of each data in the target data set; Determining, according to data characteristic information corresponding to each data contained in the target data set, a call efficiency value when each data contained in the target data set is stored according to the initial storage position; Taking maximizing the call efficiency value as the optimization goal, the initial storage position of each data in the target data set is adjusted to determine the storage position of each data in the target data set when the computing node executes the model training task of the target model.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method described in any one of claims 1 to 7 is implemented.

Images