Sample data processing method, device, equipment and storage medium

Abstract

The embodiments of the present application provide a method, apparatus, device and storage medium for processing sample data, including: obtaining a training task and a meta-information sequence corresponding to the training task, the meta-information sequence including a number of meta-information, the meta-information being used to index the corresponding sample data; traversing the meta-information sequence to determine a preset number of target meta-information; pre-storing the target sample data corresponding to the target meta-information, and using the previously pre-stored target sample data to perform the training task, and when the previously pre-stored target sample data is used up, returning to the step of traversing the meta-information sequence to determine a preset number of target meta-information, and expelling the previously pre-stored target sample data. Applying the embodiments of the present application, the sample data to be used when executing the training task is pre-stored according to the meta-information sequence, and the sample data that has been used is expelled, and only a small amount of sample data is pre-stored to meet the needs of executing the training task, so as to save the resource usage of the cache system.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method and device for processing sample data, an electronic device, and a storage medium. Background Art

[0002] In recent years, with the development of heterogeneous computing devices, more and more heterogeneous computing devices with stronger computing power have emerged, further accelerating the machine learning training process. However, the acceleration of data processing capabilities has put forward higher requirements on the data access speed of the program, and the computing and storage separation architecture adopted on the cloud further limits the data access speed. Therefore, data access speed has gradually become the main performance bottleneck of machine learning training programs.

[0003] To solve the above problems, the distributed cache solution on the computing side is currently widely used to accelerate data access: by caching the data in the storage system in the computing environment, the machine learning training jobs running in the computing environment can obtain the required data with lower latency and higher bandwidth. Since machine learning often undergoes multiple rounds of training, data cache can be reused between multiple rounds, improving the efficiency of the machine learning training process.

[0004] However, as the size of the dataset used grows, the cache system faces greater challenges: on the one hand, in order to achieve the maximum data access acceleration, the cache system needs to cache the data in the entire dataset, which means that the cache system needs to occupy a large amount of storage resources in the computing environment. On the other hand, affected by the eviction policy of the cache system, if the cache system cannot completely cache the data in the entire dataset, the data access efficiency of the machine learning training job will be significantly reduced compared to the case of full caching. Summary of the invention

[0005] An embodiment of the present application provides a method for processing sample data to solve the problem that the cache system occupies a large amount of storage resources in the computing environment, and the data access efficiency of machine learning training jobs is low when the cache system cannot completely cache the data in the entire data set.

[0006] Correspondingly, an embodiment of the present application also provides a sample data processing device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above problems, the present application discloses a method for processing sample data, which includes:

[0008] Obtaining a training task and a meta information sequence corresponding to the training task, wherein the meta information sequence includes a plurality of meta information, and the meta information is used to index the corresponding sample data;

[0009] Traversing the meta information sequence to determine a preset number of target meta information;

[0010] Pre-storing target sample data corresponding to the target meta information, and using the previously pre-stored target sample data to perform the training task;

[0011] When the previously stored target sample data is used up, the step of traversing the meta information sequence to determine a preset number of target meta information is returned to, and the previously stored target sample data is expelled.

[0012] Optionally, it also includes:

[0013] Obtaining a machine learning job and meta information corresponding to sample data in a sample data set required for learning and training of the machine learning job;

[0014] Randomly generate a total sequence of meta information using meta information corresponding to sample data in the sample data set;

[0015] The total meta information sequence is split into multiple meta information sequences, and based on the multiple meta information sequences, the machine learning job is split into multiple training tasks; wherein the multiple training tasks are executed in parallel.

[0016] Optionally, it also includes:

[0017] Get other machine learning jobs that use the sample dataset;

[0018] Based on the multiple meta-information sequences, the other machine learning jobs are split into multiple other training tasks; wherein the other training tasks are executed synchronously with the training tasks corresponding to the same meta-information sequence.

[0019] Optionally, it also includes:

[0020] Determining the time taken to perform the training task using the previously stored target sample data;

[0021] Determine the shortest time taken to execute the training task using the pre-stored target sample data in history;

[0022] Calculate the abnormal time according to the shortest time and the abnormal value parameter;

[0023] When the time is greater than the abnormal time, the preset number is increased.

[0024] Optionally, when the time is greater than the abnormal time, increasing the preset number includes:

[0025] When the time is greater than the abnormal time, detecting the number of times the step of traversing the meta information sequence and determining the preset number of target meta information is performed by adding the preset number of system time to the current system time last time;

[0026] When the number is greater than the preset number, the preset number is increased.

[0027] Optionally, it is applied to a Kubernetes cluster, where the Kubernetes cluster is deployed with working nodes and a distributed cache system, where the working nodes are used to execute the training tasks, and the distributed cache system is used to pre-store target sample data.

[0028] The present application also discloses a sample data processing device, the device comprising:

[0029] A sequence acquisition module, used to acquire a training task and a meta information sequence corresponding to the training task, wherein the meta information sequence includes a plurality of meta information, and the meta information is used to index the corresponding sample data;

[0030] A sequence traversal module, used for traversing the meta information sequence to determine a preset number of target meta information;

[0031] A task execution module, used to pre-store target sample data corresponding to the target meta-information, and use the previously pre-stored target sample data to execute the training task;

[0032] The data expulsion module is used to return to the step of traversing the meta information sequence to determine a preset number of target meta information when the previously stored target sample data is used up, and to expel the previously stored target sample data at the same time.

[0033] Optionally, it also includes:

[0034] A job acquisition module, used to acquire a machine learning job and meta information corresponding to sample data in a sample data set required for learning and training of the machine learning job;

[0035] A sequence generation module, used to randomly generate a total sequence of metadata using metadata corresponding to sample data in the sample data set;

[0036] A job splitting module is used to split the total meta-information sequence into multiple meta-information sequences, and split the machine learning job into multiple training tasks based on the multiple meta-information sequences; wherein the multiple training tasks are executed in parallel.

[0037] Optionally, it also includes:

[0038] A job acquisition module is further used to acquire other machine learning jobs using the sample data set;

[0039] A job splitting module is used to split the other machine learning jobs into multiple other training tasks based on multiple metadata sequences; wherein the other training tasks are executed synchronously with the training tasks corresponding to the same metadata sequence.

[0040] Optionally, it also includes:

[0041] A time determination module, used to determine the time taken to execute the training task using the previously stored target sample data;

[0042] The time determination module is further used to determine the shortest time taken to execute the training task using the pre-stored target sample data in history;

[0043] A time calculation module, used to calculate the abnormal time according to the shortest time and the abnormal value parameter;

[0044] The quantity increasing module is used to increase the preset quantity when the time is greater than the abnormal time.

[0045] Optionally, the quantity increasing module includes:

[0046] A times determination submodule, for, when the time is greater than the abnormal time, detecting the number of times the step of traversing the meta-information sequence and determining the preset number of target meta-information is performed by adding the preset number of system times to the current system time last time;

[0047] The quantity increasing submodule is used to increase the preset quantity when the number is greater than the preset number.

[0048] Optionally, it is applied to a Kubernetes cluster, where the Kubernetes cluster is deployed with working nodes and a distributed cache system, where the working nodes are used to execute the training tasks, and the distributed cache system is used to pre-store target sample data.

[0049] Optionally, the Kubernetes cluster is deployed with a machine learning framework, and a data reading module of the machine learning framework is replaced by a Dataset Indexing Service component to maintain a metadata sequence corresponding to the training task through the Dataset Indexing Service component.

[0050] An embodiment of the present application also discloses an electronic device, including: a processor; and a memory, on which executable code is stored. When the executable code is executed, the processor executes a method for processing sample data as described in one or more of the embodiments of the present application.

[0051] The embodiments of the present application also disclose one or more machine-readable media on which executable codes are stored. When the executable codes are executed, the processor executes the method for processing sample data as described in one or more of the embodiments of the present application.

[0052] Compared with the prior art, the embodiments of the present application have the following advantages:

[0053] In an embodiment of the present application, a training task and a meta information sequence corresponding to the training task are obtained, the meta information sequence includes several meta information, and the meta information is used to index the corresponding sample data; the meta information sequence is traversed to determine a preset number of target meta information; the target sample data corresponding to the target meta information is pre-stored, and the target sample data pre-stored previously is used to perform the training task. When the target sample data pre-stored previously is used up, the step of traversing the meta information sequence and determining the preset number of target meta information is returned, and the target sample data pre-stored previously is expelled. Applying the embodiment of the present application, according to the order of the meta information in the meta information sequence, the sample data to be used when the training task is to be executed is pre-stored first, and the sample data that has been used is expelled. As long as a small amount of sample data is pre-stored, the needs of executing the training task can be met to save the resource usage of the cache system. At the same time, the sample data to be used when the training task is to be executed is pre-stored according to the meta information sequence, which avoids the situation where the sample data cannot be hit when the training task is executed, and solves the performance bottleneck caused by slow data access. In addition, the process of expulsion and pre-storage is carried out synchronously with the execution of the training task, and the whole process runs in a pipeline manner to shorten the time of executing the training task.

[0054] In addition, after obtaining the metadata corresponding to the sample data required for the training task, the metadata is used to randomly generate a metadata sequence. The random ordering of the metadata in the metadata sequence ensures the randomness of the use of sample data in the process of executing the training task according to the metadata sequence, so as to prevent the learning model obtained by executing the training task from overfitting. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flowchart of a method for processing sample data in accordance with an embodiment of the present application;

[0056] Figure 2 is a flowchart of another example data processing method embodiment of the present application;

[0057] Figure 3is a flowchart of a step of adjusting the number of pre-stored sample data in an embodiment of the present application;

[0058] Figure 4 is a schematic diagram of a modified embodiment of a machine learning programming framework of the present application;

[0059] Figure 5 This is a schematic diagram of a framework of a Kubernetes cluster implementation example of the present application;

[0060] Figure 6 It is one of the framework schematic diagrams of a sample data processing embodiment of the present application;

[0061] Figure 7 This is the second schematic diagram of a framework of a sample data processing embodiment of the present application;

[0062] Figure 8 It is a structural block diagram of a device embodiment of sample data of the present application;

[0063] Fig. 9 It is a schematic diagram of the structure of a device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0065] Thanks to the advancement of data storage and data collection technology in the past few years, methods such as machine learning that learn expert knowledge from data have become a common method for solving computer cognitive problems. Among them, machine learning has demonstrated a strong ability to solve practical problems in various fields. Machine learning methods mainly include two processes: training and inference. Among them, the machine learning training process needs to learn the correct correlation between data and data from a large-scale data set to obtain "experience", while the machine learning inference process judges the newly emerging data based on "experience".

[0066] When actually performing machine learning training, heterogeneous computing devices such as GPUs are often used to achieve parallel acceleration of data processing. In recent years, with the research and development of heterogeneous computing devices themselves, more and more heterogeneous computing devices with stronger computing power have emerged, further accelerating the machine learning training process. The acceleration of data processing capabilities has put forward higher requirements on the data access speed of the program, and the architecture of computing and storage separation adopted on the cloud further limits the data access speed. Therefore, data access speed has gradually become the main performance bottleneck of machine learning training programs. In order to solve the above problems, the existing technology proposes the following solutions:

[0067] Using Alluxio and other popular cache technologies in the industry, distributed caching can be implemented on the computing side. Distributed caching can integrate the storage resources of each node to provide a larger cache pool to cache the entire data set required for machine learning training. However, it requires a lot of storage resources. As the size of the data set grows, more and more storage resources are required. Especially in multi-tenant scenarios, multiple machine learning training jobs require different data sets at the same time, which will put a greater burden on the distributed cache system.

[0068] Optimize the data access process within machine learning programming frameworks such as PyTorch and Tensorflow. For example, when a data access bottleneck occurs, Data Echoing improves the resource utilization of computing devices by replaying previously used data. However, such optimization essentially modifies the semantics of machine learning training and may potentially affect the effectiveness of machine learning methods.

[0069] The data pre-storage capabilities provided by the PyTorch and Tensorflow frameworks can also help alleviate the performance impact caused by data access bottlenecks. However, the pre-storage process depends on the machine learning training program written by the user. Before the program runs, the user cannot predict whether there will be a data access bottleneck in advance, so it is impossible to reasonably configure the pre-storage function provided by the framework.

[0070] In order to solve the above problems, the core idea of ​​this application is to replace the existing data reading components by injection at the framework layer of the application container when deploying machine learning applications in a cluster, and to manage the pre-storage and eviction of sample data through a unified service, so as to achieve the purpose of improving the application running speed under the premise of lower data space.

[0071] 1. Introduce the Dataset Indexing Service component and automatically replace the data reading modules of machine learning frameworks such as TensorFlow and PyTorch to control the order of sample data access for machine learning training jobs.

[0072] 2. The Dataset Indexing Service component manages the pre-storage and eviction of sample data in the cache system according to the data access sequence characteristics of different applications, making efficient use of the cache.

[0073] 3. Dynamically control the pre-storage of sample data according to the machine learning training speed to ensure that the machine learning training process is as unaffected by data access bottlenecks as possible.

[0074] The present application provides a method and device for processing sample data, which are described in detail in the following embodiments. First, the terms involved in one or more embodiments of the present application are explained.

[0075] Kubernetes: is an open source system for automating the deployment, scaling, and management of containerized applications.

[0076] Machine learning: It is essentially the use of data to solve computer cognitive problems, including knowledge understanding, information processing and even prediction.

[0077] Remote storage system: refers to cloud storage or storage servers that are far away from the computing side and are used to store training sample data sets.

[0078] Distributed cache: refers to storing remotely stored data on machines close to users or applications in a distributed environment or system to reduce the delay of remote data transmission, allowing users and applications to quickly access the desired data.

[0079] GPU: Graphics Processing Unit, similar to CPU, except that GPU is designed specifically to perform complex mathematical and geometric calculations and is widely used in artificial intelligence.

[0080] Worker node: A computer node with a GPU or other resources for learning and training.

[0081] Meta-information: also called metadata, is data about data, mainly information describing data properties, used to support functions such as indicating storage location, historical data, resource search, file records, etc. It is considered an electronic catalog. In order to achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby achieving the purpose of assisting data retrieval.

[0082] Reference Figure 1 , is a flowchart of a method for processing sample data of the present application, comprising the following steps:

[0083] Step 101, obtaining a training task and a meta information sequence corresponding to the training task, wherein the meta information sequence includes a plurality of meta information, and the meta information is used to index to the corresponding sample data.

[0084] The method is applied to the Kubernetes cluster, which is deployed with machine learning frameworks such as TensorFlow and PyTorch. The Dataset Indexing Service component is introduced, and the data reading module of the machine learning framework is automatically replaced. At the same time, the Kubernetes cluster is deployed with working nodes and a distributed cache system.

[0085] Specifically, after obtaining the training task and the metadata sequence corresponding to the training task, the training task is scheduled to the working node, and the metadata sequence corresponding to the training task is maintained by the Dataset Indexing Service component. The metadata sequence is randomly generated after the Dataset Indexing Service component obtains the metadata corresponding to the sample data required for the training task. The random ordering of the metadata in the metadata sequence ensures the randomness of the use (consumption) of the sample data in the process of executing the training task according to the metadata sequence, so as to prevent the overfitting of the learning model obtained by executing the training task.

[0086] Step 102, traverse the meta information sequence to determine a preset number of target meta information.

[0087] Specifically, when a working node needs to consume (use) sample data to perform a training task, the working node will request the sample data from the Dataset Indexing Service component. The Dataset Indexing Service component will traverse the metadata sequence in sequence, determine the preset number of target metadata, and return it to the working node. For example, if the preset number is 2 and the metadata sequence is 1, 3, 6, 5, 4, 5, then the preset number of target metadata is determined to be 1 and 3.

[0088] Step 103: pre-store target sample data corresponding to the target meta information, and use the previously pre-stored target sample data to perform the training task.

[0089] The remote storage system stores sample data corresponding to the meta-information in the meta-information sequence.

[0090] Specifically, the target sample data corresponding to the target metadata is pulled from the remote storage system and pre-stored in the distributed cache system. At the same time, the target sample data pre-stored last time is used to perform the training task. For example, the metadata sequence is 1, 3, 6, 5, 4, 5, and the distributed cache system has pre-stored sample data 1 and 3. Therefore, the working node requests sample data from the DatasetIndexing Service component. The Dataset Indexing Service component traverses the metadata sequence in sequence and determines that the target metadata is 6 and 5. Then the distributed cache system starts to pre-store sample data 6 and 5. At the same time, the working node obtains sample data 1 and 3 from the distributed cache system to perform the training task.

[0091] Step 104, when the previously stored target sample data is used up, return to the step of traversing the meta information sequence to determine a preset number of target meta information, and simultaneously expel the previously stored target sample data.

[0092] Specifically, when the previously stored target sample data is used (consumed), the working node will request sample data from the Dataset Indexing Service component again. The Dataset Indexing Service component will traverse the metadata sequence again in order to determine the new preset number of target metadata. It should be noted that the metadata corresponding to the previously stored sample data will not be determined as the target metadata. For example, if the preset number is 2 and the metadata sequence is 1, 3, 6, 5, 4, 5, then the previously determined preset number of target metadata is 1 and 3. The metadata sequence is traversed again, starting from metadata 6, and the preset number of target metadata is determined to be 6 and 5. At the same time, the previously stored target sample data in the distributed cache system is expelled.

[0093] In the embodiment of the present application, a training task and a meta information sequence corresponding to the training task are obtained, the meta information sequence includes a number of meta information, and the meta information is used to index the corresponding sample data; the meta information sequence is traversed to determine a preset number of target meta information; the target sample data corresponding to the target meta information is pre-stored, and the target sample data pre-stored last time is used to perform the training task. When the target sample data pre-stored last time is used up, the step of traversing the meta information sequence and determining a preset number of target meta information is returned, and the target sample data pre-stored last time is expelled. Applying the embodiment of the present application, the sample data to be used (consumed) for executing the training task is pre-stored according to the meta information sequence, and the sample data that has been used is expelled. As long as a small amount of sample data is pre-stored, the needs of executing the training task can be met, and all the sample data required for executing the training task do not need to be pre-stored, so as to save the resource usage of the cache system. At the same time, according to the meta information sequence, the sample data to be used when executing the training task is pre-stored first, avoiding the situation where the sample data cannot be hit when executing the training task, and solving the performance bottleneck caused by slow data access. In addition, the process of eviction and pre-storage is carried out synchronously with the execution of training tasks, and the entire process runs in a pipeline manner, shortening the time for executing training tasks.

[0094] In addition, the metadata sequence is randomly generated after the Dataset Indexing Service component obtains the metadata corresponding to the sample data required for the training task. The random ordering of the metadata in the metadata sequence ensures the randomness of the use (consumption) of the sample data in the process of executing the training task according to the metadata sequence, so as to prevent the learning model obtained by executing the training task from overfitting.

[0095] Based on the above embodiment, an optional embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the optional embodiment.

[0096] In one embodiment of the present application, refer to Figure 2 , shows a flowchart of another example data processing method embodiment of the present application, including the following steps:

[0097] Step 201, obtaining a machine learning job and meta information corresponding to sample data in a sample data set required for learning and training of the machine learning job.

[0098] Specifically, obtain the machine learning job and the metadata corresponding to the sample data in the sample data set that the machine learning job needs to use for learning and training, wherein the metadata corresponding to the sample data in the sample data set is stored in the distributed cache system, and the metadata corresponding to the sample data is obtained from the distributed cache system through the Dataset Indexing Service component.

[0099] Step 202: randomly generate a total sequence of meta information using the meta information corresponding to the sample data in the sample data set.

[0100] Among them, based on the performance analysis of the program for machine learning training on large-scale data sets, it can be found that the random data access order of machine learning training is the root cause of the significant reduction in data access efficiency. The data access order during machine learning training is completely random. If the cache system cannot cache all the data, cache eviction strategies such as LRU will evict the data that has not been used recently, causing the machine learning training to miss the cache in the next data access.

[0101] Specifically, after the Dataset Indexing Service component obtains the metadata corresponding to the sample data from the distributed cache system, the metadata is used to randomly generate a total sequence of metadata. The machine learning job can access (use) the sample data according to the order of the metadata in the total sequence of metadata for learning and training.

[0102] In the embodiment of the present application, the metadata corresponding to the sample data is randomly generated into a total sequence of metadata in advance. When the machine learning operation is trained, the sample data can be accessed according to the order of the metadata in the total sequence of metadata, which ensures the randomness of the sample data used during the learning and training process to prevent the obtained learning model from overfitting. At the same time, the problem of cache misses in the subsequent sample data access during learning and training is solved.

[0103] Step 203: split the total meta-information sequence into multiple meta-information sequences, and split the machine learning job into multiple training tasks based on the multiple meta-information sequences; wherein the multiple training tasks are executed in parallel.

[0104] Specifically, the total sequence of metadata is split into multiple sequences of metadata through the Dataset Indexing Service component, and the machine learning job is split into multiple training tasks based on the multiple sequences of metadata, and the multiple training tasks are respectively scheduled to different working nodes for parallel execution. The steps for the working node to execute the training task have been described above and will not be repeated here.

[0105] In an embodiment of the present application, a machine learning job is split into multiple training tasks based on multiple metadata sequences, and the multiple training tasks are scheduled to different work nodes for parallel execution, which can shorten the time spent on learning and training of the machine learning job.

[0106] In one embodiment of the present application, it also includes: obtaining other machine learning jobs using the sample data set; splitting the other machine learning jobs into multiple other training tasks based on multiple metadata sequences; wherein the other training tasks are executed synchronously with the training tasks corresponding to the same metadata sequence.

[0107] Specifically, other machine learning jobs that use the sample data set (of the machine learning job) are obtained. Since the machine learning job and the other machine learning jobs use the same sample data set for learning and training, the machine learning job and the other machine learning jobs can be coordinated in a centralized manner to share the same total sequence of metadata. Therefore, other machine learning jobs can be split into multiple other training tasks based on multiple metadata sequences, and the multiple other training tasks can be scheduled to different work nodes for execution. The training tasks and other training tasks corresponding to the same metadata sequence can be executed synchronously. During the execution process, the training tasks and other training tasks use the same sample data, so that the sample data in the distributed cache system can be reused by multiple training tasks, thereby improving cache utilization.

[0108] In an embodiment of the present application, a machine learning job is split into multiple training tasks based on multiple metadata sequences, and the multiple training tasks are scheduled to different work nodes for parallel execution, which can shorten the time spent on learning and training of the machine learning job.

[0109] In addition, based on multiple metadata sequences, machine learning jobs are split into multiple training tasks, and other machine learning jobs are split into multiple other training tasks. The training tasks and other training tasks corresponding to the same metadata sequence can be executed synchronously. During the execution process, the training tasks and other training tasks use the same sample data, so that the sample data in the distributed cache system can be reused by multiple training tasks, thereby improving cache utilization.

[0110] Based on the above embodiment, an optional embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the optional embodiment.

[0111] In one embodiment of the present application, refer to Figure 3 , shows a flowchart of a step of adjusting the number of pre-stored sample data in the present application, including the following steps:

[0112] Step 301: Determine the time taken to execute the training task using the previously stored target sample data.

[0113] Step 302: Determine the shortest time taken to execute the training task using pre-stored target sample data in history.

[0114] Step 303: Calculate the abnormal time according to the shortest time and the abnormal value parameter.

[0115] Step 304: When the time is greater than the abnormal time, increase the preset number.

[0116] Among them, the amount of pre-stored sample data in the distributed cache system is related to the speed of using sample data when executing training tasks and the speed of pulling sample data from the remote storage system. If the amount of pre-stored sample data is too small, some sample data will still not be in the distributed cache system when they are used, and cache misses will lead to a decrease in the efficiency of sample data access. If the amount of pre-stored sample data is too large, it will occupy too many storage resources in the distributed cache system.

[0117] Specifically, determine the time taken to execute the training task using the previously pre-stored preset number of target sample data, and the shortest time taken to execute the training task using the pre-stored target sample data in history, multiply the shortest time by the outlier parameter to obtain the outlier time; when the time taken to execute the training task using the previously pre-stored preset number of target sample data is greater than the outlier time, it means that some sample data is still not in the distributed cache system when being used, and it is necessary to increase the preset number, that is, increase the number of pre-stored sample data in the distributed cache system, to meet the needs of executing the training task.

[0118] In one embodiment of the present application, when the time is greater than the abnormal time, increasing the preset number includes: when the time is greater than the abnormal time, detecting the system time of the last time the preset number was increased to the current system time to execute the step of traversing the metadata sequence to determine the number of times a preset number of target metadata is obtained; when the number is greater than the preset number, increasing the preset number.

[0119] Among them, for a machine learning job, the amount of sample data consumed in each worker node request interval is fixed, and the operation logic for this part of the training samples is fixed, so the overall operation time is also very stable, and only a small range of time fluctuations may occur. Therefore, the amount of pre-stored sample data in the distributed cache system is increased to an appropriate amount, and no adjustment is required within a certain period of time.

[0120] Specifically, when the time taken to execute a training task using a preset number of target sample data pre-stored in the previous time is greater than the abnormal time, the last time the system time of increasing the preset number of system time to the current system time is detected to execute the traversal of the metadata sequence, and the number of steps of obtaining the preset number of target metadata is determined. When the number is greater than the preset number, the preset number is increased, that is, the number of sample data pre-stored in the distributed cache system is increased. When the number is less than or equal to the preset number, there is no need to increase the preset number, that is, there is no need to increase the number of sample data pre-stored in the distributed cache system.

[0121] It should also be noted that the speed at which machine learning jobs consume sample data is relatively stable. Therefore, once the amount of sample data pre-stored in the distributed cache system is expanded to an appropriate size, it usually does not need to be reduced.

[0122] The following formula is used to determine whether the preset quantity needs to be increased:

[0123]

[0124]

[0125] Where: t i It indicates the time interval between the current request from the working node and the last request from the working node, that is, the time spent on executing the training task using the previously stored target sample data; T i min It represents the minimum value of the request interval in the past time period, that is, the shortest time spent in the history of using the pre-stored target sample data to execute the training task; α is the outlier parameter; i represents the number of steps of detecting the last time the preset number of system time is increased to the current system time to traverse the metadata sequence and determine the preset number of target metadata; P is the stable value parameter, that is, the preset number of times.

[0126] When t i Greater than α*T i min , and when i is greater than P, the preset number is increased, that is, the number of pre-stored sample data in the distributed cache system is increased.

[0127] In an embodiment of the present application, by detecting the time spent on executing a training task using a preset number of target sample data, the number of pre-stored sample data in the distributed cache system can be adjusted to avoid the situation where the number of pre-stored sample data is too small, resulting in some sample data still not being in the distributed cache system when being used, and cache misses leading to reduced sample data access efficiency, while an excessive number of pre-stored sample data occupies too many storage resources in the distributed cache system.

[0128] In order to better understand the embodiments of the present application, the method for processing sample data is exemplified below, but it should be understood that the embodiments of the present application are not limited thereto.

[0129] Reference Figure 4 , showing a schematic diagram of a modified embodiment of a machine learning programming framework of the present application, the technical solution in the present application requires modification of the underlying data access logic of the machine learning programming framework (such as PyTorch, Tensorflow), while users often use the standard version of the machine learning programming framework, which does not contain the code logic to implement the technical solution of the present application. When the user submits a job, the Service Auto Injector component (hereinafter referred to as the Injector component) will create a Dataset Indexing Service component (hereinafter referred to as the Service component) in the cluster, and the Service component will start to pre-store sample data after it is started. Then, the Injector component injects InitContainer (a container used for initialization work) into the machine learning job submitted by the user. The InitContainer uses the image defined in this solution, and the image contains code logic changes to the machine learning programming framework. InitContainer will start before the machine learning job submitted by the user. When InitContainer starts, the code logic that needs to be changed will be overwritten to the corresponding position of the user image, realizing user-unaware logic replacement. When the user-defined machine learning job is started, its data access process will be carried out according to the workflow described below.

[0130] It should be noted that without introducing the Dataset Indexing Service component, by modifying the data access logic of the machine learning framework, machine learning can also manage the cache of the distributed cache system while accessing data.

[0131] Reference Figure 5 , showing a schematic diagram of the framework of a Kubernetes cluster embodiment of the present application, before the machine learning training job starts, the Service component obtains the metadata of all training sample data (sample data) in the entire data set from the distributed cache system. The order in which the data of the samples consumed during the machine learning training process should be completely random. In order to prevent the machine learning model from overfitting, when the Service component obtains the metadata of the training samples, it will first shuffle them to generate a random total sequence of metadata, and split the total sequence of metadata into multiple sequences of metadata to correspond to the training tasks of multiple machine learning training workers;

[0132] When any machine learning training worker needs to consume data to perform training tasks, the machine learning training worker requests training sample data from the Service component. The Service component traverses the metadata sequence corresponding to the machine learning training worker and selects the metadata corresponding to the unused training sample data to return.

[0133] The machine learning training worker reads the corresponding sample data from the distributed cache system based on the returned metadata to perform the training task.

[0134] Reference Figure 6 , shows one of the framework schematic diagrams of a sample data processing embodiment of the present application, the distributed cache system has pre-stored sample data 1, 3, 5, when the Service component returns the meta information 3, 5 to the machine learning training Worker, the machine learning training Worker reads the sample data 3, 5 from the distributed cache system. Figure 7 At this time, when the machine learning training Worker requests training samples from the Service component, the sliding window (dashed window) moves right, and the new metadata 6 and 4 move into the sliding window. At this time, the Service component immediately performs the pre-storage operation of the sample data 6 and 4. While the sliding window moves right, the metadata 3 and 5 of the sliding window are moved out. Since the sample data corresponding to the metadata 3 and 5 have just been requested and are being consumed by the machine learning training Worker for model training (executing training tasks), the Service component will not immediately evict the sample data 3 and 5 from the distributed cache system. When the machine learning training Worker requests sample data from the Service component again, it indicates that the previously requested sample data 3 and 5 have been consumed. At this time, the Service component can evict the sample data 3 and 5.

[0135] In the embodiment of the present application, the sample data that has been consumed will not be used again, so the Service component instructs the distributed cache system to evict these sample data to save the resource usage of the cache system. At the same time, new metadata is moved into the sliding window, and the Service component instructs the distributed cache system to pre-store the sample data corresponding to the metadata from the remote storage system to the distributed cache system, so that the machine learning training Worker can hit the cache when it needs this part of the sample data, so as to avoid the performance bottleneck caused by slow data access. The above-mentioned eviction and pre-storage process is carried out synchronously with the model training process of the machine learning training Worker, and the entire process runs in a pipeline manner to shorten the overall machine learning training time.

[0136] It should be noted that, for the method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present application are not limited by the described order of actions, because according to the embodiments of the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present application.

[0137] On the basis of the above embodiments, this embodiment further provides a sample data processing device, which is applied to electronic devices such as terminal devices and servers.

[0138] Reference Figure 8 , shows a structural block diagram of a device embodiment of a sample data of the present application, which may specifically include the following modules:

[0139] A sequence acquisition module 801 is used to acquire a training task and a meta information sequence corresponding to the training task, wherein the meta information sequence includes a plurality of meta information, and the meta information is used to index to the corresponding sample data;

[0140] A sequence traversal module 802 is used to traverse the meta information sequence and determine a preset number of target meta information;

[0141] The task execution module 803 is used to pre-store the target sample data corresponding to the target meta information, and use the previously pre-stored target sample data to execute the training task;

[0142] The data expulsion module 804 is used to return to the step of traversing the meta information sequence to determine a preset number of target meta information when the previously stored target sample data is used up, and to expel the previously stored target sample data.

[0143] In one embodiment of the present application, it also includes:

[0144] A job acquisition module, used to acquire a machine learning job and meta information corresponding to sample data in a sample data set required for learning and training of the machine learning job;

[0145] A sequence generation module, used to randomly generate a total sequence of metadata using metadata corresponding to sample data in the sample data set;

[0146] A job splitting module is used to split the total meta-information sequence into multiple meta-information sequences, and split the machine learning job into multiple training tasks based on the multiple meta-information sequences; wherein the multiple training tasks are executed in parallel.

[0147] In one embodiment of the present application, it also includes:

[0148] A job acquisition module is further used to acquire other machine learning jobs using the sample data set;

[0149] A job splitting module is used to split the other machine learning jobs into multiple other training tasks based on multiple metadata sequences; wherein the other training tasks are executed synchronously with the training tasks corresponding to the same metadata sequence.

[0150] In one embodiment of the present application, it also includes:

[0151] A time determination module, used to determine the time taken to perform the training task using the previously stored target sample data;

[0152] The time determination module is further used to determine the shortest time taken to execute the training task using the pre-stored target sample data in history;

[0153] A time calculation module, used to calculate the abnormal time according to the shortest time and the abnormal value parameter;

[0154] The quantity increasing module is used to increase the preset quantity when the time is greater than the abnormal time.

[0155] In one embodiment of the present application, the quantity increasing module includes:

[0156] A times determination submodule, for, when the time is greater than the abnormal time, detecting the number of times the step of traversing the meta-information sequence and determining the preset number of target meta-information is performed by adding the preset number of system times to the current system time last time;

[0157] The quantity increasing submodule is used to increase the preset quantity when the number is greater than the preset number.

[0158] In one embodiment of the present application, it is applied to a Kubernetes cluster, and the Kubernetes cluster is deployed with working nodes and a distributed cache system. The working nodes are used to execute the training tasks, and the distributed cache system is used to pre-store target sample data.

[0159] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of each method step in the embodiment of the present application.

[0160] The present application embodiment provides one or more machine-readable media on which instructions are stored, and when executed by one or more processors, an electronic device executes one or more of the methods described in the above embodiments. In the present application embodiment, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0161] The embodiments of the present disclosure may be implemented as a device configured as desired using any appropriate hardware, firmware, software, or any combination thereof, and the device may include electronic devices such as terminal devices and servers (clusters). Fig. 9 An exemplary apparatus 900 that can be used to implement various embodiments described in this application is schematically illustrated.

[0162] For one embodiment, Fig. 9 An exemplary apparatus 900 is shown having one or more processors 902, a control module (chip set) 904 coupled to at least one of the processor(s) 902, a memory 906 coupled to the control module 904, a non-volatile memory (NVM) / storage device 908 coupled to the control module 904, one or more input / output devices 910 coupled to the control module 904, and a network interface 912 coupled to the control module 904.

[0163] The processor 902 may include one or more single-core or multi-core processors, and the processor 902 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.). In some embodiments, the device 900 can be used as a terminal device, server (cluster), etc. described in the embodiments of the present application.

[0164] In some embodiments, the apparatus 900 may include one or more computer-readable media (e.g., memory 906 or NVM / storage device 908) having instructions 914 and one or more processors 902 combined with the one or more computer-readable media and configured to execute the instructions 914 to implement a module to perform the actions described in the present disclosure.

[0165] For one embodiment, the control module 904 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 902 and / or any suitable device or component in communication with the control module 904 .

[0166] The control module 904 may include a memory controller module to provide an interface to the memory 906. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0167] The memory 906 may be used, for example, to load and store data and / or instructions 914 for the device 900. For one embodiment, the memory 906 may include any suitable volatile memory, such as a suitable DRAM. In some embodiments, the memory 906 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0168] For one embodiment, control module 904 may include one or more input / output controllers to provide an interface to NVM / storage device 908 and input / output device(s) 910 .

[0169] For example, NVM / storage 908 may be used to store data and / or instructions 914. NVM / storage 908 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable non-volatile storage device(s) (e.g., one or more hard disk drives (HDDs), one or more compact disk (CD) drives, and / or one or more digital versatile disk (DVD) drives).

[0170] NVM / storage device 908 may include storage resources that are physically part of the device on which apparatus 900 is installed, or it may be accessible to the device without being part of the device. For example, NVM / storage device 908 may be accessed via (one or more) input / output devices 910 over a network.

[0171] (One or more) input / output devices 910 may provide an interface for the apparatus 900 to communicate with any other appropriate device, and the input / output device 910 may include a communication component, an audio component, a sensor component, etc. The network interface 912 may provide an interface for the apparatus 900 to communicate through one or more networks, and the apparatus 900 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, for example, accessing a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof for wireless communication.

[0172] For one embodiment, at least one of the processor(s) 902 may be packaged together with the logic of one or more controllers (e.g., a memory controller module) of the control module 904. For one embodiment, at least one of the processor(s) 902 may be packaged together with the logic of one or more controllers of the control module 904 to form a system-in-package (SiP). For one embodiment, at least one of the processor(s) 902 may be integrated on the same die with the logic of one or more controllers of the control module 904. For one embodiment, at least one of the processor(s) 902 may be integrated on the same die with the logic of one or more controllers of the control module 904 to form a system-on-chip (SoC).

[0173] In various embodiments, the device 900 may be, but is not limited to, a terminal device such as a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the device 900 may have more or fewer components and / or a different architecture. For example, in some embodiments, the device 900 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0174] Among them, the main control chip can be used as a processor or control module in the detection device, sensor data, location information, etc. are stored in a memory or NVM / storage device, the sensor group can be used as an input / output device, and the communication interface may include a network interface.

[0175] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0176] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0177] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable xxxx terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable xxxx terminal device generate instructions for implementing the processes in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0178] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0179] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0180] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0181] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0182] The above is a detailed introduction to a method and device for processing sample data, an electronic device and a storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for processing sample data, characterized in that: The method comprises: Obtaining a training task and a meta information sequence corresponding to the training task, wherein the meta information sequence includes a plurality of meta information, and the meta information is used to index the corresponding sample data; Traversing the meta information sequence to determine a preset number of target meta information; Pre-storing target sample data corresponding to the target meta information, and using the previously pre-stored target sample data to perform the training task; When the previously stored target sample data is used up, returning to the step of traversing the meta information sequence to determine a preset number of target meta information, and simultaneously expelling the previously stored target sample data; Determining the time taken to perform the training task using the previously stored target sample data; Determine the shortest time taken to execute the training task using the pre-stored target sample data in history; Calculate the abnormal time according to the shortest time and the abnormal value parameter; When the time is greater than the abnormal time, the preset number is increased.

2. The method according to claim 1, characterized in that Also includes: Obtaining a machine learning job and meta information corresponding to sample data in a sample data set required for learning and training of the machine learning job; Randomly generate a total sequence of meta information using meta information corresponding to sample data in the sample data set; The total meta information sequence is split into multiple meta information sequences, and based on the multiple meta information sequences, the machine learning job is split into multiple training tasks; wherein the multiple training tasks are executed in parallel.

3. The method according to claim 2, characterized in that Also includes: Get other machine learning jobs that use the sample dataset; Based on the multiple meta-information sequences, the other machine learning jobs are split into multiple other training tasks; wherein the other training tasks are executed synchronously with the training tasks corresponding to the same meta-information sequence.

4. The method according to claim 1, characterized in that When the time is greater than the abnormal time, increasing the preset number includes: When the time is greater than the abnormal time, detecting the number of times the step of traversing the meta information sequence and determining the preset number of target meta information is performed by adding the preset number of system time to the current system time last time; When the number is greater than the preset number, the preset number is increased.

5. The method according to claim 1, characterized in that: Applied to a Kubernetes cluster, the Kubernetes cluster is deployed with working nodes and a distributed cache system, the working nodes are used to execute the training tasks, and the distributed cache system is used to pre-store target sample data.

6. The method according to claim 5, characterized in that The Kubernetes cluster is deployed with a machine learning framework, and the data reading module of the machine learning framework is replaced by a Dataset Indexing Service component to maintain the metadata sequence corresponding to the training task through the Dataset Indexing Service component.

7. A sample data processing device, characterized in that: The device comprises: A sequence acquisition module, used to acquire a training task and a meta information sequence corresponding to the training task, wherein the meta information sequence includes a plurality of meta information, and the meta information is used to index to the corresponding sample data; A sequence traversal module, used for traversing the meta information sequence to determine a preset number of target meta information; A task execution module, used to pre-store target sample data corresponding to the target meta-information, and use the previously pre-stored target sample data to execute the training task; A data expulsion module, configured to return to the step of traversing the meta information sequence to determine a preset number of target meta information and to expel the previously stored target sample data when the previously stored target sample data is used up; A time determination module, used to determine the time taken to perform the training task using the previously stored target sample data; The time determination module is further used to determine the shortest time taken to execute the training task using the pre-stored target sample data in history; A time calculation module, used to calculate the abnormal time according to the shortest time and the abnormal value parameter; The quantity increasing module is used to increase the preset quantity when the time is greater than the abnormal time.

8. The device according to claim 7, characterized in that Also includes: A job acquisition module, used to acquire a machine learning job and meta information corresponding to sample data in a sample data set required for learning and training of the machine learning job; A sequence generation module, used to randomly generate a total sequence of metadata using metadata corresponding to sample data in the sample data set; A job splitting module is used to split the total meta-information sequence into multiple meta-information sequences, and split the machine learning job into multiple training tasks based on the multiple meta-information sequences; wherein the multiple training tasks are executed in parallel.

9. An electronic device, characterized in that: include: processor; and A memory having executable codes stored thereon, which, when executed, enables the processor to execute the method for processing sample data as described in any one of claims 1 to 6.

10. One or more machine-readable media having executable codes stored thereon, which, when executed, enable a processor to execute the method for processing sample data as claimed in any one of claims 1 to 6.

Images