A system, method and apparatus for exchanging data in machine learning

Abstract

Embodiments of the present application provide a data exchange system, method and device in machine learning, the system and method comprising: determining a target storage strategy according to a data exchange amount of a data exchange window in which a target data set is located and the topological relationship information; sending the target storage strategy to an upstream processing server to enable the upstream processing server to store the target data set to a storage area corresponding to a unified multi-level storage server; and sending storage distribution information of the target data set to a downstream processing server to enable the downstream processing server to acquire the target data set and perform a downstream machine learning step to write the target data set. The method improves the efficiency and stability of data exchange in a distributed machine learning pipeline.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a data exchange system, method and apparatus for machine learning. Background Technology

[0002] A distributed machine learning pipeline comprises multiple machine learning steps, including feature engineering, model training, model evaluation, and model prediction. During the execution of machine learning tasks, data exchange occurs between different steps in the pipeline; that is, downstream steps need to read the dataset output by upstream steps and then process the data based on that dataset. Currently, data exchange between the steps of a machine learning pipeline is primarily achieved through the following methods:

[0003] Method 1: Data exchange is performed by writing data to disk. That is, the upstream server node that performs the upstream steps writes the dataset to the file system, and the downstream server node that performs the downstream steps reads the corresponding dataset from the file system.

[0004] Method 2: Data exchange is conducted via RPC (Remote Procedure Call Protocol). The upstream server node that executes the upstream steps provides the endpoint, and the downstream server node that executes the downstream steps obtains the corresponding dataset through network communication.

[0005] However, the first data exchange method uses a file system for data exchange, which generates a large amount of disk I / O during the execution of machine learning pipeline tasks, severely impacting the execution efficiency of the machine learning pipeline. The second data exchange method uses network communication for data exchange. This method requires the front-end server node to remain running after completing its task to provide endpoints. However, maintaining a running state indefinitely prolongs the lifecycle of the server node, thereby reducing the stability of the server node's task execution. Summary of the Invention

[0006] The purpose of this invention is to provide a data exchange system, method, and apparatus for machine learning, so as to improve the execution efficiency and stability of data exchange in machine learning pipelines.

[0007] In a first aspect, embodiments of the present invention provide a data exchange system for machine learning, comprising: a global control storage server, a unified multi-level storage server, and multiple processing servers; each of the processing servers corresponds to a machine learning step in the machine learning pipeline;

[0008] An upstream processing server is used to execute the target machine learning step in the machine learning pipeline, obtain the target dataset output by the target machine learning step, and send a storage strategy request for the target dataset to the global control storage server; wherein, the upstream processing server is the processing server corresponding to the target learning step;

[0009] The global control storage server is used to acquire the topological relationship information between each machine learning step in the machine learning pipeline, and after receiving the storage strategy request, determine the target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topological relationship information, and send the target storage strategy to the upstream processing server; wherein, the data exchange window is predetermined by the global control storage server based on the topological relationship information, and each data exchange window includes at least one dataset exchanged between machine learning steps;

[0010] The upstream processing server is also configured to store the target dataset to the storage area corresponding to the unified multi-level storage server according to the received target storage strategy.

[0011] A downstream processing server is configured to, after the downstream machine learning step of the target machine learning step is triggered, obtain the storage distribution information of the target dataset from the global control storage server, obtain the target dataset from the unified multi-level storage server according to the storage distribution information, and execute the operation of writing the target dataset by the downstream machine learning step; wherein, the downstream processing server is the processing server corresponding to the downstream machine learning step.

[0012] Optionally, the global control storage server is specifically used to acquire the input-output relationship information between each machine learning step in the machine learning pipeline, and to determine the topological relationship information between each machine learning step based on the input-output relationship information.

[0013] Optionally, the system may also include a pipeline scheduling server;

[0014] The pipeline scheduling server is used to determine the topological relationship information between the machine learning steps in the machine learning pipeline based on the input-output relationship information between the machine learning steps in the machine learning pipeline, send the topological relationship information to the global control storage server, and trigger the upstream processing server to execute the target machine learning step in the machine learning pipeline based on the topological relationship information.

[0015] The global control storage server is specifically used to receive the topology information sent by the pipeline scheduling server;

[0016] The pipeline scheduling server is also used to trigger the downstream processing server to execute the downstream machine learning step after the target machine learning step is completed.

[0017] Optionally, the global control storage server is further configured to determine the data exchange window based on the topology information using the following formula:

[0018]

[0019] Where n is the number of data exchange windows in the machine learning pipeline, W n W is the nth data exchange window in the machine learning pipeline. n-i Let C0 be the n-th data exchange window in the machine learning pipeline, G0 be the set of vertices with an out-degree of 0 in G0, and G0 be the topological relationship information of each dataset in the machine learning pipeline. i Let G0 be the set of datasets with an out-degree of 0 before the i-th topology pruning. i-1 Let G0 be a set of datasets with an out-degree of 0 before the (i-1)th topology pruning. Topology pruning is defined as deleting datasets with an out-degree of 0 based on their topological relationships. The out-degree of a dataset is the number of times it is used as input to downstream machine learning steps, as determined by the topological relationships. If a dataset is not used as input to any downstream machine learning step, then its out-degree is 0.

[0020] Optionally, the unified multi-level storage server includes a first storage area, a second storage area, and a third storage area; the storage space of the first storage area is smaller than the storage space of the second storage area, and the storage space of the second storage area is smaller than the storage space of the third storage area.

[0021] The global control storage server is specifically used to allocate the remaining storage space of the first storage area to the target dataset according to the data exchange window where the target dataset is located, and according to the proportion of data exchange volume of each dataset in the data exchange window; if the first storage area cannot completely store the target dataset, the storage space of the second storage area is allocated to the remaining data of the target dataset; if neither the first storage area nor the second storage area can completely store the target dataset, the storage space of the third storage area is allocated to the remaining data of the target dataset, and the proportion of storage area allocated to the target dataset is used as the target storage strategy.

[0022] Optionally, the global control storage server is specifically used to allocate the remaining storage space of the first storage area to the target dataset according to the proportion of data exchange volume of each dataset in the data exchange window where the target dataset is located, using the following formula:

[0023]

[0024] in, i-1 d is the set of datasets in the data exchange window where the target dataset is located that have not been allocated storage space in the first storage area. j For datasets without allocated storage space in this data exchange window, C i This is the collection of datasets for which storage space in the first storage region has been allocated within the data exchange window. For dataset d j The maximum remaining storage space in the allocated first storage region, λ out (d j ) for dataset d j The number of times it is used for data exchange, λ out (d α ) for dataset d α The number of times it is used for data exchange, E(d) j ) for dataset d j The size of the data, d α For set C i-1 One of the datasets, E(d) α ) for dataset d α The size of the data, d β For set C i One of the datasets, Q real (d β ) is for dataset d β The allocated storage space for the first storage region, Q total This refers to the storage space in the first storage area.

[0025] Optionally, the global control storage server is further configured to acquire the data volume of each historical exchange dataset of the machine learning pipeline; and based on the data volume of the historical exchange dataset, calculate the data exchange volume of each dataset in the data exchange window where the target dataset is located.

[0026] Secondly, embodiments of the present invention provide a method for exchanging data in machine learning, applied to a global control storage server in a data exchange system for machine learning, the system further including a unified multi-level storage server and multiple processing servers, the method comprising:

[0027] Obtain the topological relationship information between each machine learning step in the machine learning pipeline;

[0028] The system receives a storage strategy request for the target dataset from an upstream processing server and determines a target storage strategy based on the data exchange volume of the data exchange window containing the target dataset and the topology information. The target dataset is the dataset output by the target machine learning step after the upstream processing server executes the target machine learning pipeline. The data exchange window is predetermined by the global control storage server based on the topology information, and each data exchange window includes at least one dataset exchanged between machine learning steps. The upstream processing server is the processing server corresponding to the target learning step.

[0029] The target storage policy is sent to the upstream processing server so that the upstream processing server stores the target dataset in the storage area corresponding to the unified multi-level storage server according to the received target storage policy.

[0030] After the downstream machine learning step of the target machine learning step is triggered, the storage distribution information of the target dataset is sent to the downstream processing server so that: the downstream processing server obtains the target dataset from the unified multi-level storage server according to the storage distribution information and executes the operation of writing the target dataset by the downstream machine learning step; wherein, the downstream processing server is the processing server corresponding to the downstream machine learning step.

[0031] Optionally, before obtaining the topological relationship information between each machine learning step in the machine learning pipeline, the method further includes:

[0032] Obtain the input-output relationship information between each machine learning step in the machine learning pipeline;

[0033] The step of obtaining the input-output relationship information between each machine learning step in the machine learning pipeline includes:

[0034] The topological relationship information between each machine learning step is determined based on the input-output relationship information.

[0035] Optionally, the system may also include a pipeline scheduling server;

[0036] The acquisition of topological relationship information between various machine learning steps in the machine learning pipeline includes:

[0037] The pipeline scheduling server obtains the topological relationship information between the various machine learning steps; wherein the topological relationship information is determined by the pipeline scheduling server based on the input-output relationship information between the various machine learning steps in the machine learning pipeline.

[0038] Optionally, before determining the target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topology information, the method further includes:

[0039] Based on the aforementioned topological relationship information, the data exchange window is determined using the following formula:

[0040]

[0041] Where n is the number of data exchange windows in the machine learning pipeline, W n W is the nth data exchange window in the machine learning pipeline. n-i Let C0 be the n-th data exchange window in the machine learning pipeline, G0 be the set of vertices with an out-degree of 0 in G0, and G0 be the topological relationship information of each dataset in the machine learning pipeline. i Let G0 be the set of datasets with an out-degree of 0 before the i-th topology pruning. i-1 Let G0 be a set of datasets with an out-degree of 0 before the (i-1)th topology pruning. Topology pruning is defined as deleting datasets with an out-degree of 0 based on their topological relationships. The out-degree of a dataset is the number of times it is used as input to downstream machine learning steps, as determined by the topological relationships. If a dataset is not used as input to any downstream machine learning step, then its out-degree is 0.

[0042] Optionally, the unified multi-level storage server includes a first storage area, a second storage area, and a third storage area; the storage space of the first storage area is smaller than the storage space of the second storage area, and the storage space of the second storage area is smaller than the storage space of the third storage area.

[0043] The step of determining the target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topology information includes:

[0044] Based on the data exchange window where the target dataset is located, and according to the proportion of data exchange volume of each dataset in the data exchange window, the remaining storage space of the first storage area is allocated to the target dataset; if the first storage area cannot completely store the target dataset, the storage space of the second storage area is allocated to the remaining data of the target dataset; if neither the first storage area nor the second storage area can completely store the target dataset, the storage space of the third storage area is allocated to the remaining data of the target dataset, and the proportion of storage areas allocated to the target dataset is used as the target storage strategy.

[0045] Optionally, the step of allocating the remaining storage space of the first storage area to the target dataset according to the proportion of data exchange volume of each dataset in the data exchange window where the target dataset is located includes:

[0046] The remaining storage space in the first storage area is allocated to the target dataset using the following formula, based on the proportion of data exchange volume of each dataset within the data exchange window where the target dataset is located:

[0047]

[0048] in, i-1 d is the set of datasets in the data exchange window where the target dataset is located that have not been allocated storage space in the first storage area. j For datasets without allocated storage space in this data exchange window, C i This is the collection of datasets for which storage space in the first storage region has been allocated within the data exchange window. For dataset d j The maximum remaining storage space in the allocated first storage region, λ out (d j ) for dataset d j The number of times it is used for data exchange, λ out (d α ) for dataset d α The number of times it is used for data exchange, E(d) j ) for dataset d j The size of the data, d α For set C i-1 One of the datasets, E(d) α ) for dataset d α The size of the data, d β For set C i One of the datasets, Q real (d β ) is for dataset d β The allocated storage space for the first storage region, Q total This refers to the storage space in the first storage area.

[0049] Optionally, before allocating the remaining storage space of the first storage area to the target dataset according to the proportion of data exchange volume of each dataset in the data exchange window where the target dataset is located, the method further includes:

[0050] Obtain the data volume of each historical exchange dataset in the machine learning pipeline;

[0051] Based on the data volume of the historical exchange dataset, calculate the data exchange volume of each dataset in the data exchange window where the target dataset is located.

[0052] Thirdly, embodiments of the present invention provide a data exchange device for machine learning, applied to a global control storage server in a data exchange system for machine learning, the system further including a unified multi-level storage server and multiple processing servers, the device comprising:

[0053] The topology acquisition module is used to acquire topology relationship information between various machine learning steps in the machine learning pipeline;

[0054] A storage strategy determination module is used to receive a storage strategy request for the target dataset sent by an upstream processing server, and determine a target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topology information; wherein, the target dataset is the dataset output by the target machine learning step obtained after the upstream processing server executes the target machine learning pipeline; the data exchange window is predetermined by the global control storage server based on the topology information, and each data exchange window includes at least one dataset exchanged between machine learning steps; the upstream processing server is the processing server corresponding to the target learning step;

[0055] A storage policy sending module is used to send the target storage policy to the upstream processing server, so that the upstream processing server stores the target dataset in the storage area corresponding to the unified multi-level storage server according to the received target storage policy.

[0056] The distribution information sending module is used to send the storage distribution information of the target dataset to the downstream processing server after the downstream machine learning step of the target machine learning step is triggered, so that: the downstream processing server obtains the target dataset from the unified multi-level storage server according to the storage distribution information and executes the operation of writing the target dataset by the downstream machine learning step; wherein, the downstream processing server is the processing server corresponding to the downstream machine learning step.

[0057] Optionally, the device further includes:

[0058] The input-output relationship module is used to obtain the input-output relationship information between each machine learning step in the machine learning pipeline;

[0059] The topology relationship acquisition module is specifically used to determine the topology relationship information between each machine learning step based on the input-output relationship information.

[0060] Optionally, the system may also include a pipeline scheduling server;

[0061] The topology acquisition module is specifically used to acquire topology relationship information between the various machine learning steps from the pipeline scheduling server; wherein, the topology relationship information is determined by the pipeline scheduling server based on the input-output relationship information between the various machine learning steps in the machine learning pipeline.

[0062] Optionally, the device further includes:

[0063] The data exchange window determination module is used to determine the data exchange window based on the topology information using the following formula:

[0064]

[0065] Where n is the number of data exchange windows in the machine learning pipeline, W n W is the nth data exchange window in the machine learning pipeline. n-i Let C0 be the n-th data exchange window in the machine learning pipeline, G0 be the set of vertices with an out-degree of 0 in G0, and G0 be the topological relationship information of each dataset in the machine learning pipeline. i Let G0 be the set of datasets with an out-degree of 0 before the i-th topology pruning. i-1 Let G0 be a set of datasets with an out-degree of 0 before the (i-1)th topology pruning. Topology pruning is defined as deleting datasets with an out-degree of 0 based on their topological relationships. The out-degree of a dataset is the number of times it is used as input to downstream machine learning steps, as determined by the topological relationships. If a dataset is not used as input to any downstream machine learning step, then its out-degree is 0.

[0066] Optionally, the unified multi-level storage server includes a first storage area, a second storage area, and a third storage area; the storage space of the first storage area is smaller than the storage space of the second storage area, and the storage space of the second storage area is smaller than the storage space of the third storage area.

[0067] The storage strategy determination module is specifically used to allocate the remaining storage space of the first storage area to the target dataset according to the data exchange window where the target dataset is located, and according to the proportion of data exchange volume of each dataset in the data exchange window; if the first storage area cannot completely store the target dataset, allocate the storage space of the second storage area to the remaining data of the target dataset; if neither the first storage area nor the second storage area can completely store the target dataset, allocate the storage space of the third storage area to the remaining data of the target dataset, and use the proportion of storage areas allocated to the target dataset as the target storage strategy.

[0068] Optionally, the storage strategy determination module is specifically used to allocate the remaining storage space of the first storage area to the target dataset according to the proportion of data exchange volume of each dataset in the data exchange window where the target dataset is located, using the following formula:

[0069]

[0070] in, i-1 d is the set of datasets in the data exchange window where the target dataset is located that have not been allocated storage space in the first storage area. j For datasets without allocated storage space in this data exchange window, C i This is the collection of datasets for which storage space in the first storage region has been allocated within the data exchange window. For dataset d j The maximum remaining storage space in the allocated first storage region, λ out (d j ) for dataset d j The number of times it is used for data exchange, λ out (d α ) for dataset d α The number of times it is used for data exchange, E(d) j ) for dataset d j The size of the data, d α For set C i-1 One of the datasets, E(d) α ) for dataset d α The size of the data, d β For set C i One of the datasets, Q real (d β ) is for dataset d β The allocated storage space for the first storage region, Q total This refers to the storage space in the first storage area.

[0071] Optionally, the device further includes:

[0072] The data volume acquisition module is used to acquire the data volume of each historical exchange dataset in the machine learning pipeline; based on the data volume of the historical exchange dataset, it calculates the data exchange volume of each dataset in the data exchange window where the target dataset is located.

[0073] Fourthly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0074] Memory, used to store computer programs;

[0075] When a processor executes a program stored in memory, it implements any of the steps described in the second aspect above.

[0076] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described in the second aspect above.

[0077] Beneficial effects of the embodiments of the present invention:

[0078] The machine learning data exchange system provided in this embodiment of the invention allows the global control storage server to determine the target storage strategy based on the data exchange volume and topology information of the data exchange window where the target dataset is located. Then, based on the target storage strategy, the target dataset is stored in the storage area corresponding to the unified multi-level storage server, thus orderly realizing data exchange between various machine learning steps in the machine learning pipeline and effectively improving the efficiency of distributed machine learning pipeline data exchange. Furthermore, the machine learning data exchange system provided in this embodiment of the invention does not require the front-end server node to remain running after completing its task, thereby improving the stability of data exchange.

[0079] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0080] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0081] Figure 1A schematic diagram of a data exchange system in machine learning provided in an embodiment of the present invention;

[0082] Figure 2 A schematic diagram illustrating the input-output relationship between machine learning steps in a distributed machine learning pipeline;

[0083] Figure 3 Another schematic diagram of the structure of a data exchange system in machine learning provided in this embodiment of the invention;

[0084] Figure 4 This is a schematic diagram illustrating the topological relationships between various machine learning steps in a distributed machine learning pipeline.

[0085] Figure 5a The topology graph obtained after the first topology pruning of the DAG graph of the distributed machine learning pipeline;

[0086] Figure 5b The topology graph obtained after the second topology pruning of the DAG graph of the distributed machine learning pipeline;

[0087] Figure 6 A schematic diagram of the data exchange window in a distributed machine learning pipeline;

[0088] Figure 7 A diagram illustrating the storage space size of each storage area in a unified multi-level storage service server;

[0089] Figure 8 A schematic diagram of data exchange in a distributed machine learning pipeline's data exchange window;

[0090] Figure 9 A diagram illustrating the amount of data exchanged by the data exchange window in a distributed machine learning pipeline.

[0091] Figure 10 This diagram illustrates the allocation of distributed memory storage resources for the data exchange window in a distributed machine learning pipeline.

[0092] Figure 11 A flowchart of a data exchange method in machine learning provided by an embodiment of the present invention;

[0093] Figure 12 A schematic diagram of a data exchange device in machine learning provided in an embodiment of the present invention;

[0094] Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0095] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0096] Since using file systems for data exchange in distributed machine learning pipelines affects the execution efficiency of machine learning pipelines, and using network communication for data exchange in distributed machine learning pipelines reduces the stability of server nodes executing tasks, this invention provides a data exchange system, method, apparatus, electronic device, computer-readable storage medium, and computer program product for machine learning in order to improve the execution efficiency and stability of data exchange in machine learning pipelines.

[0097] The following is a description of the technical concepts involved in the embodiments of the present invention:

[0098] A distributed machine learning pipeline includes multiple machine learning steps such as feature engineering, model training, model evaluation, model prediction, and model deployment, and each machine learning step is scheduled to run on a different server. In this embodiment of the invention, the server running the machine learning steps is referred to as a processing server.

[0099] Data exchange: There is data exchange between different machine learning steps in a machine learning pipeline. That is, the downstream machine learning step needs to read the dataset output by the upstream machine learning step as its input dataset. The datasets that are exchanged between the upstream and downstream machine learning steps can be called exchange datasets.

[0100] Data exchange window: As the distributed machine learning pipeline executes each machine learning step, storage resources are occupied and released, forming a data exchange window in the distributed machine learning pipeline.

[0101] The following section first introduces the data exchange system for machine learning provided in the embodiments of the present invention.

[0102] Figure 1 This is a schematic diagram of a data exchange system in machine learning provided by an embodiment of the present invention, such as... Figure 1 As shown, it includes: a global control storage server 110, a unified multi-level storage server 120, and multiple processing servers 130; each of the processing servers corresponds to a machine learning step in the machine learning pipeline;

[0103] An upstream processing server 130 is used to execute the target machine learning step in the machine learning pipeline, obtain the target dataset output by the target machine learning step, and send a storage strategy request for the target dataset to the global control storage server; wherein, the upstream processing server is the processing server corresponding to the target learning step;

[0104] The global control storage server 110 is used to acquire the topological relationship information between each machine learning step in the machine learning pipeline, and after receiving the storage strategy request, determines the target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topological relationship information, and sends the target storage strategy to the upstream processing server 130; wherein, the data exchange window is predetermined by the global control storage server 110 based on the topological relationship information, and each data exchange window includes at least one dataset exchanged between machine learning steps;

[0105] The upstream processing server 130 is also configured to store the target dataset to the storage area corresponding to the unified multi-level storage server 120 according to the received target storage strategy.

[0106] Downstream processing server 130 is configured to, after the downstream machine learning step of the target machine learning step is triggered, obtain the storage distribution information of the target dataset from the global control storage server 110, obtain the target dataset from the unified multi-level storage server 120 according to the storage distribution information, and execute the operation of writing the target dataset by the downstream machine learning step; wherein, the downstream processing server is the processing server corresponding to the downstream machine learning step.

[0107] The machine learning data exchange system provided in this embodiment of the invention allows the global control storage server to determine the target storage strategy based on the data exchange volume and topology information of the data exchange window where the target dataset is located. Then, based on the target storage strategy, the target dataset is stored in the storage area corresponding to the unified multi-level storage server, thus orderly realizing data exchange between various machine learning steps in the machine learning pipeline and effectively improving the efficiency of distributed machine learning pipeline data exchange. Furthermore, the machine learning data exchange system provided in this embodiment of the invention does not require the front-end server node to remain running after completing its task, thereby improving the stability of data exchange.

[0108] In this embodiment of the invention, GCS (Global Control Store) is used to store the configuration information of the machine learning pipeline, the datasets and allocation strategies output by each machine learning step, and the amount of data used to evaluate each dataset.

[0109] UMS (Unified Multi-level Storage) provides a unified storage interface and supports multi-level storage types, from distributed memory to remote distributed file systems.

[0110] PSJ (Pipeline Step Job) involves different computing frameworks, such as Spark (a cluster computing framework that distributes large dataset computing tasks across multiple servers to improve efficient in-memory computing), Flink (an open-source stream processing framework that executes arbitrary streaming data programs in a data-parallel and pipelined manner), Tensorflow (a system that transmits complex data structures to artificial intelligence neural networks for analysis and processing), and PyTorch (an open-source deep learning framework).

[0111] In this embodiment of the invention, each processing server corresponds to a machine learning step in the machine learning pipeline, and each processing server executes the corresponding machine learning step. For example... Figure 1 As shown, if processing server 1 corresponds to machine learning step A in the machine learning pipeline, processing server 2 corresponds to machine learning step B in the machine learning pipeline, and processing server 3 corresponds to machine learning step C in the machine learning pipeline, then processing server 1 can be used to execute machine learning step A, processing server 2 can be used to execute machine learning step B, and processing server 3 can be used to execute machine learning step C.

[0112] In one possible implementation, the global control storage server can specifically be used to acquire input-output relationship information between various machine learning steps in the machine learning pipeline, and determine the topological relationship information between the various machine learning steps based on the input-output relationship information. Specifically, Figure 2 This is a schematic diagram illustrating the input-output relationships between different machine learning steps in a distributed machine learning pipeline. Figure 2In the distributed machine learning pipeline, Pipeline Steps A through F represent machine learning steps A through F, respectively. Datasets a through F represent the output datasets from the processing servers corresponding to machine learning steps A and F, respectively, after each step. For example, Dataset a is the output dataset from the processing server corresponding to machine learning step A after executing step A, and it also serves as the input dataset for machine learning steps C and D. Similarly, Dataset b is the output dataset from the processing server corresponding to machine learning step B after executing step B, and it also serves as the input dataset for machine learning step D.

[0113] like Figure 2 As shown, the global control storage server can obtain the input and output information of each machine learning step from each processing server, obtain the input-output relationship information between each machine learning step in the machine learning pipeline, and then determine the topological relationship information between each machine learning step based on the input-output relationship information.

[0114] In one possible implementation, Figure 3 This is another schematic diagram of the structure of the data exchange system in machine learning provided in the embodiments of the present invention, such as... Figure 3 As shown, the data exchange system in machine learning also includes a pipeline scheduling server 310;

[0115] The pipeline scheduling server 310 is used to determine the topological relationship information between the machine learning steps in the machine learning pipeline based on the input-output relationship information between the machine learning steps in the machine learning pipeline, send the topological relationship information to the global control storage server, and trigger the upstream processing server to execute the target machine learning step in the machine learning pipeline based on the topological relationship information.

[0116] The global control storage server 110 is specifically used to receive the topology information sent by the pipeline scheduling server 310;

[0117] The pipeline scheduling server 310 is also used to trigger the downstream processing server to execute the downstream machine learning step after the target machine learning step is completed.

[0118] Specifically, the pipeline scheduling server can obtain the input and output information of each machine learning step from the various processing servers in the machine learning pipeline, thereby obtaining the input-output relationship information between the various machine learning steps in the machine learning pipeline. For example... Figure 2 As shown, the pipeline scheduling server can obtain the following: The distributed machine learning pipeline includes machine learning steps A through F. Machine learning step A outputs dataset a, machine learning step B outputs dataset b, datasets a and b together serve as input to machine learning step D, machine learning step D outputs dataset d based on datasets a and b, dataset a serves as input to machine learning step C, machine learning step C outputs dataset c based on dataset a, dataset c and d together serve as input to machine learning step E, machine learning step E outputs dataset e based on datasets c and d, dataset e serves as input to machine learning step F, machine learning step F outputs dataset f based on dataset e, and dataset f is the final output dataset of the distributed machine learning pipeline, which no longer participates in data exchange.

[0119] like Figure 2As shown, the pipeline scheduling server registers the resource configuration of the distributed machine learning pipeline and the input and output information of each step with the global control storage server. After successful registration, the pipeline scheduling server first triggers the execution of Pipeline Step A according to the topology of each machine learning step. The configuration information registered by the pipeline scheduling server can include information about each storage area of ​​the multi-level storage server backbone. When Pipeline Step A performs a write operation on Dataset a, it registers Dataset a with the global control storage server and requests a storage policy for Dataset a. The global control storage server receives the registration of Dataset a and calculates the storage policy for Dataset a based on the overall resource configuration of the distributed machine learning pipeline, remaining storage resources, the topology of the machine learning steps, and the estimated data volume. Based on the returned storage policy, Pipeline Step A writes Dataset a to the unified multi-level storage server and notifies the global control storage server of the successful write. After Pipeline Step A completes execution, the pipeline scheduling server triggers the execution of Pipeline Step C. When Pipeline Step C performs the dataset a read operation, it obtains the storage distribution information of the dataset a from the global control storage server and the specific data of the dataset a from the unified multi-level storage server.

[0120] In this embodiment of the invention, a global control storage server provides dynamic storage strategy allocation based on the storage resources of a unified multi-level storage server, thereby achieving global optimization of data exchange in a distributed machine learning pipeline.

[0121] Figure 4 This is a diagram illustrating the topological relationships between the various machine learning steps in a distributed machine learning pipeline, such as... Figure 4 As shown, the pipeline scheduling server can... Figure 2 The machine learning pipeline shown obtains input and output information for each machine learning step from each processing server, and then determines the appropriate steps based on the input and output information of each machine learning step. Figure 4 The diagram shows the topological relationships between the various machine learning steps in the machine learning pipeline. Figure 4This diagram illustrates the Directed Acyclic Graph (DAG) of the datasets exchanged between various machine learning steps. The distributed machine learning pipeline includes machine learning step AF. Since machine learning step F is the final step in the distributed machine learning pipeline, and the dataset f output by machine learning step F does not participate in data exchange, the DAG does not include datasets that do not participate in data exchange. Figure 4 The topological relationship diagram between the various machine learning steps in the distributed machine learning pipeline illustrates the relationship between the various datasets involved in data exchange. Figure 4 In the diagram, Dataset a-Dataset e represent the datasets output by the processing servers corresponding to machine learning step A and machine learning step E, respectively, after each server has completed its corresponding machine learning step. The direction of the arrows for the datasets indicates their out-degree. The out-degree of a dataset is the number of times that dataset is used as input to downstream machine learning steps, as determined by the topology information. If a dataset is not used as input to any downstream machine learning step, its out-degree is 0. Figure 4 In the topology diagram shown, the number of times the dataset Dataset e is used as input to the downstream machine learning step is 0, therefore the out-degree of the dataset Dataset e is 0.

[0122] Furthermore, the pipeline scheduling server can trigger the upstream processing server to execute the target machine learning step in the machine learning pipeline based on the topology information. Typically, the pipeline scheduling server triggers the corresponding processing server to execute the corresponding machine learning step sequentially according to the execution order of each machine learning step in the machine learning pipeline. For example, if based on the topology information... Figure 2 If machine learning step A needs to be executed first, the pipeline scheduling server will first trigger the processing server corresponding to machine learning step A to execute machine learning step A. After machine learning step A is completed, the pipeline scheduling server can also trigger the processing server corresponding to the downstream step of machine learning step A—machine learning step C—to execute machine learning step C.

[0123] In one possible implementation, the global control storage server is further configured to determine the data exchange window based on the topology information using the following formula:

[0124]

[0125] Where n is the number of data exchange windows in the machine learning pipeline, W n W is the nth data exchange window in the machine learning pipeline. n-iLet C0 be the n-th data exchange window in the machine learning pipeline, G0 be the set of vertices with an out-degree of 0 in G0, and G0 be the topological relationship information of each dataset in the machine learning pipeline. i Let G0 be the set of datasets with an out-degree of 0 before the i-th topology pruning. i-1 Let G0 be a set of datasets with an out-degree of 0 before the (i-1)th topology pruning. Topology pruning is defined as deleting datasets with an out-degree of 0 based on their topological relationships. The out-degree of a dataset is the number of times it is used as input to downstream machine learning steps, as determined by the topological relationships. If a dataset is not used as input to any downstream machine learning step, then its out-degree is 0.

[0126] For example, according to Figure 4 This refers to the DAG (Directed Acyclic Graph) of the datasets exchanged between different machine learning steps in a distributed machine learning pipeline. Figure 4 The DAG graph shown can be represented by G0. C0 is the set of datasets in G0 with out-degrees of 0, which means that set C0 includes Dataset e. Performing the first topological trimming on G0 removes the datasets Dataset e with out-degrees of 0, resulting in the trimmed topological graph denoted as G1. G1 can... Figure 5a As shown, Figure 5a Let C1 be the topology obtained after the first topology pruning of the DAG graph in the distributed machine learning pipeline, and let C1 be the set of vertices with an out-degree of 0 after the first topology pruning of G0. Then, according to... Figure 5a We can obtain set C1, which includes Dataset c and Dataset d. Performing a second topology pruning on G0 removes Dataset c and Dataset d, both with out-degrees of 0, resulting in a pruned topology graph, denoted as G2. G2 can... Figure 5b As shown, Figure 5b The topology graph obtained after the second topology pruning of the DAG graph of the distributed machine learning pipeline; Figure 5b The topology diagram shown only contains datasets Dataset a and Dataset b, and Figure 5b Datasets a and b both have an out-degree of 0; C2 is the set of vertices with an out-degree of 0 after the second topological pruning of G0, i.e., after... Figure 5a The set of vertices with an out-degree of 0 after topological pruning of the topological graph G1 shown can be obtained as set C2, which includes Dataset b and Dataset b.

[0127] The number of data exchange windows, n, in a distributed machine learning pipeline is equal to: the maximum number of topological pruning operations that can be performed on the DAG graph of the datasets exchanged between different machine learning steps in the distributed machine learning pipeline, plus 1. Figure 4 The DAG graph shown can only be topologically pruned twice, which yields the following result. Figure 4 The number of data exchange windows n in the distributed machine learning pipeline corresponding to the DAG shown is equal to 3.

[0128] Based on Figure 4 The diagram showing the topological relationships between the various machine learning steps in the distributed machine learning pipeline can be used to determine, based on the aforementioned data exchange window, the relationships between them. Figure 6 The three data exchange windows shown are: Figure 6 A schematic diagram of the data exchange window in a distributed machine learning pipeline:

[0129] If n = 3, C0 includes Dataset e, C1 includes Dataset c and Dataset d, and C2 includes Dataset a and Dataset b, then the data exchange windows Window1 (W1), Window2 (W2), and Window3 (W3) respectively include:

[0130]

[0131] like Figure 6 As shown, the data exchange window Window1 includes datasets Dataset a, Dataset b, Dataset c, and Dataset d; the data exchange window Window2 includes datasets Dataset c, Dataset d, and Dataset e; and the data exchange window Window3 includes datasets Dataset a, Dataset b, Dataset c, and Dataset d.

[0132] In this embodiment of the invention, based on the aforementioned formula for determining the data exchange window, various data exchange windows can be determined, including the datasets interacting between different machine learning steps in a distributed machine learning pipeline. Then, the datasets within each data exchange window can be treated as a whole, and storage resources from a unified multi-level storage server can be allocated to each dataset within that data exchange window. For example, Figure 6The data exchange window Window2 shown includes datasets Dataset c, Dataset d, and Dataset e. For high-quality storage resources such as the first storage area of ​​the unified multi-level storage server, high-quality storage resources can be allocated to datasets Dataset c, Dataset d, and Dataset e as a whole, instead of directly allocating high-quality storage resources to a single dataset.

[0133] In one possible implementation, the unified multi-level storage server includes a first storage area, a second storage area, and a third storage area; the storage space of the first storage area is smaller than that of the second storage area, and the storage space of the second storage area is smaller than that of the third storage area. Specifically, the global control storage server is used to allocate the remaining storage space of the first storage area to the target dataset according to the proportion of data exchange volume of each dataset within the data exchange window where the target dataset is located; if the first storage area cannot fully store the target dataset, the remaining data of the target dataset is allocated to the storage space of the second storage area; if neither the first nor the second storage area can fully store the target dataset, the remaining data of the target dataset is allocated to the storage space of the third storage area, and the proportion of storage areas allocated to the target dataset is used as the target storage strategy.

[0134] Storage resources are essential for data exchange between machine learning steps in a distributed machine learning pipeline, and the read / write costs of different storage resources vary significantly. Distributed memory can serve as a high-quality storage resource, supporting high-speed read / write capabilities, but its cost is relatively high, and its storage resources are relatively limited. Distributed file systems, on the other hand, are relatively inexpensive and offer relatively abundant storage resources. In this embodiment of the invention, the unified multi-level storage server includes a first storage area, a second storage area, and a third storage area; wherein the first storage area can be a memory storage area, the second storage area can be a local distributed file system storage area, and the third storage area can be a remote distributed file system storage area. Before each machine learning step in the distributed machine learning pipeline is executed, storage resources for data exchange need to be configured. Since the storage space of distributed memory and local distributed file systems is relatively limited, it is necessary to limit the amount of distributed memory and local distributed file system storage space used by the distributed machine learning pipeline during data exchange. However, remote distributed file systems have abundant storage resources, so there is no need to limit their use during data exchange. Figure 7A diagram illustrating the storage space size of each storage region in a unified multi-level storage service server is shown below. Figure 7 As shown, the storage space of Distributed Memory (i.e., the first storage area) in the unified multi-level storage server can be configured to 100GB, the storage space of the Local Distributed FileSystem (i.e., the second storage area) in the unified multi-level storage server can be configured to 2TB, and the storage space of the Remote Local Distributed FileSystem (i.e., the third storage area) in the unified multi-level storage server can be configured to 2PB. During the execution of machine learning steps in the distributed machine learning pipeline, if the storage space occupied by the exchanged dataset does not exceed 100GB, it is stored in distributed memory; if the storage space occupied by the exchanged dataset exceeds 100GB, it overflows into the local distributed file system for storage; if the storage space occupied by the exchanged dataset exceeds 2TB, the excess portion continues to overflow into the remote distributed file system for storage.

[0135] In this embodiment of the invention, by unifying multi-level storage servers and utilizing the high-performance read and write capabilities of distributed memory's superior storage resources, data exchange efficiency is improved, thereby enhancing the execution efficiency of the distributed machine learning pipeline. Furthermore, by leveraging the massive storage resources of the remote distributed file system, the jitter and overflow issues in data exchange are resolved, thus improving the stability of data exchange.

[0136] For example, Figure 8 The following is a schematic diagram of the data exchange window in a distributed machine learning pipeline. Figure 8 As shown, in the data exchange window W1 of the distributed machine learning pipeline, there are datasets Dataset a, Dataset b, Dataset c, and Dataset d. Pipeline Step A writes dataset Dataset a to a unified multi-level storage server, and Pipeline Step C reads dataset Dataset a from the unified multi-level storage server. The size of dataset Dataset a is the amount of data exchanged between Pipeline Step A and B, denoted as S(a). Therefore, the total amount of data exchanged in W1 is:

[0137] S(W1)=2S(a)+S(b)+S(c)+S(d) (1)

[0138] In W1, dataset a is exchanged simultaneously by Pipeline Step C and Pipeline Step D via a unified multi-level storage server. Therefore, the coefficient before S(a) in the above formula is the out-degree of dataset a, denoted as λ. out (a) can be used to summarize the above formula (1) to obtain formula (2):

[0139] S(W1)=λ out S(a)+S(b)+S(c)+S(d) (2)

[0140] For the data exchange window W of a distributed machine learning pipeline i If it contains datasets d1, d2, ... d m The formula for calculating the total amount of data exchanged is:

[0141]

[0142] As each machine learning step in the distributed machine learning pipeline is executed, the data exchange window continuously slides, and the amount of data exchanged in each window also changes continuously. Data exchange occurs via a unified multi-level storage server. Given the storage resources of this unified multi-level storage server, when S(W) i When the maximum value is reached, it is necessary to ensure that the high-quality storage resource of distributed memory of the unified multi-level storage service server is fully utilized. Figure 9 A diagram illustrating the amount of data exchanged by the data exchange window in a distributed machine learning pipeline, as shown below. Figure 9 As shown, it is assumed that the size of each swapped dataset is equal, such as Figure 9 As shown, this distributed machine learning pipeline has three data exchange windows W1, W2, and W3. The amount of data exchanged in each data exchange window is as follows: Figure 9 As shown: Data exchange window W1 has 2S(a)+S(b)+S(c)+S(d), that is, the total amount of data exchanged is 5; data exchange window W2 has S(c)+S(d)+S(e), that is, the total amount of data exchanged is 3; data exchange window W3 has S(e), that is, the total amount of data exchanged is 1.

[0143] like Figure 9 As shown, the total amount of data exchanged in data exchange window W1 is the largest. Therefore, the high-quality distributed memory of the unified multi-level storage server should be fully allocated to W1 first. Assuming that the data size of each dataset is the same, the proportion of distributed memory allocated to W1 by the unified multi-level storage server according to the proportion of data exchange volume is as follows:

[0144] d a :d b :d c :d d =2:1:1:1

[0145] After allocating distributed memory storage resources to the unified multi-level storage server in data exchange window W1, the window can be slid up or down to allocate distributed memory storage resources to each dataset in all data exchange windows of the distributed machine learning pipeline. When the window slides to W2, datasets Dataset a and Dataset b release their distributed memory storage resources. Since W2 only adds dataset Dataset e, all remaining distributed memory storage resources after the release are allocated to dataset Dataset e. Figure 10 This diagram illustrates the distributed memory storage resource allocation for data exchange windows in a distributed machine learning pipeline. Similarly, the distributed memory storage resource allocation for all data exchange windows in a distributed machine learning pipeline can be illustrated as follows: Figure 10 As shown.

[0146] During the sliding of the data exchange window in a distributed machine learning pipeline, the storage resources of distributed memory can be dynamically allocated in the following manner:

[0147] Assume the current data exchange window is W. n-i Based on the above formula for calculating the data exchange window, the data exchange window W can be obtained. n-i The included dataset is:

[0148] W n-i =C i-1 +C i

[0149] Data exchange window W n-i The previous data exchange window W n-(i+1) The included dataset is:

[0150] W n-(i+1) =C i+1 +C i

[0151] Among them, C i The dataset in the data exchange window does not release distributed memory storage resources during the sliding process of the data exchange window. When the data exchange window moves from W... n-(i+1) Slide to W n-i It needs to be C i-1 The dataset is used for distributed memory storage resource allocation. Data exchange window W n-i Data set d j (d j ∈Ci-1 The formula for calculating the maximum usable distributed memory storage resources is as follows:

[0152]

[0153] in, i-1 d is the set of datasets in the data exchange window where the target dataset is located that have not been allocated storage space in the first storage area. j For datasets without allocated storage space in this data exchange window, C i This is the collection of datasets for which storage space in the first storage region has been allocated within the data exchange window. For dataset d j The maximum remaining storage space in the allocated first storage region, λ out (d j ) for dataset d j The number of times it is used for data exchange, λ out (d α ) for dataset d α The number of times it was used for data exchange, d α For set C i-1 One of the datasets, d β For set C i One of the datasets, Q real (d β ) is for dataset d β The allocated storage space for the first storage region, Q total This refers to the storage space in the first storage area.

[0154] In this embodiment of the invention, during the process of sliding the previous data exchange window to the next data exchange window, the high-quality storage resources occupied by dataset b can be released in a timely manner. The released high-quality resources can be used for data exchange between subsequent machine learning steps in the distributed machine learning pipeline. That is, this embodiment of the invention can achieve fine-grained control over storage resources in a unified multi-level storage server through the data exchange window.

[0155] In one possible implementation, the global control storage server is further configured to acquire the data volume of each historical exchange dataset in the machine learning pipeline; and, based on the data volume of the historical exchange datasets, calculate the data exchange volume of each dataset within the data exchange window containing the target dataset. Specifically, the global control storage server is configured to allocate the remaining storage space of the first storage area to the target dataset using the following formula, according to the proportion of the data exchange volume of each dataset within the data exchange window containing the target dataset:

[0156]

[0157] in, i-1 d is the set of datasets in the data exchange window where the target dataset is located that have not been allocated storage space in the first storage area. j For datasets without allocated storage space in this data exchange window, C i This is the collection of datasets for which storage space in the first storage region has been allocated within the data exchange window. For dataset d j The maximum remaining storage space in the allocated first storage region, λ out (d j ) for dataset d j The number of times it is used for data exchange, λ out (d α ) for dataset d α The number of times it is used for data exchange, E(d) j ) for dataset d j The size of the data, d α For set C i-1 One of the datasets, E(d) α ) for dataset d α The size of the data, d β For set C i One of the datasets, Q real (d β ) is for dataset d β The allocated storage space for the first storage region, Q total This refers to the storage space in the first storage area.

[0158] Specifically, formula (4) above is derived under the premise that the data size of each exchanged dataset is the same. Typically, when a distributed machine learning pipeline executes a machine learning step for the first time, it cannot predict the data size of the exchanged datasets, so it can be assumed that the data size of each exchanged dataset is equal. However, during subsequent cycle scheduling executions of the distributed machine learning pipeline, the data exchange volume of each dataset within the data exchange window containing the target dataset can be calculated based on the data size of each historical exchanged dataset in the distributed machine learning pipeline. Specifically, time-series statistical analysis can be performed based on the data size of the exchanged datasets in the historical execution records of the distributed machine learning pipeline to predict the data size of each exchanged dataset. For example, a unified multi-level storage server records the actual data size exchanged for each dataset during the execution of the distributed machine learning pipeline. Table 1 below shows the time series of the actual data size exchanged for each dataset:

[0159] Table 1: Time series of the actual amount of data exchanged in the dataset

[0160]

[0161]

[0162] When analyzing the actual amount of data exchanged across different datasets based on historical time series statistics, it's necessary to consider not only the most recent time series characteristics but also those over a given period. Based on Table 1, weighted averages were calculated for the last 3, 7, 14, and 30 data points, with weights of 0.4, 0.3, 0.2, and 0.1 respectively. The final weighted average statistics are shown in Table 2 below.

[0163] Table 2: Actual Data Volume Exchanged in Time-Series Weighted Average Statistical Data Sets

[0164]

[0165] By using a time-series weighted average to calculate the actual amount of data exchanged between each dataset, we can predict the actual amount of data exchanged between each dataset during the next execution of the distributed machine learning pipeline, denoted as E(d). j Considering that the actual amount of data exchanged in the datasets is not the same, the data exchange window W can be finally obtained according to formula (4). n-i Data set d j (d j ∈C i-1 The formula for calculating the maximum distributed memory storage resources that can be used is the above formula (5).

[0166] This invention addresses the instability of data exchange caused by data volume fluctuations by unifying the massive storage resources of a remote distributed file system in a multi-level storage server. Furthermore, during the execution of the distributed machine learning pipeline, dynamic storage strategies are allocated through data exchange windows and data volume evaluation to achieve global optimization of data exchange in the distributed machine learning pipeline, thereby improving the stability and execution efficiency of data exchange and enhancing the performance of machine learning.

[0167] Corresponding to the data exchange system in machine learning described above, this embodiment of the invention also provides a data exchange method in machine learning, which is applied to a global control storage server in a data exchange system for machine learning. The system further includes a unified multi-level storage server and multiple processing servers. Figure 11 A flowchart of a data exchange method in machine learning provided by an embodiment of the present invention is shown below. Figure 11 As shown, the method includes:

[0168] S1101, obtain the topological relationship information between each machine learning step in the machine learning pipeline.

[0169] S1102, receive a storage strategy request for the target dataset sent by the upstream processing server, and determine the target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topology information.

[0170] Wherein, the target dataset is the dataset output by the target machine learning step obtained after the upstream processing server executes the target machine learning pipeline; the data exchange window is predetermined by the global control storage server according to the topology information, and each data exchange window includes at least one dataset exchanged between machine learning steps; the upstream processing server is the processing server corresponding to the target learning step.

[0171] S1103, the target storage policy is sent to the upstream processing server so that the upstream processing server stores the target dataset in the storage area corresponding to the unified multi-level storage server according to the received target storage policy.

[0172] S1104, after the downstream machine learning step of the target machine learning step is triggered, the storage distribution information of the target dataset is sent to the downstream processing server so that: the downstream processing server obtains the target dataset from the unified multi-level storage server according to the storage distribution information and executes the operation of writing the target dataset by the downstream machine learning step.

[0173] The downstream processing server is the processing server corresponding to the downstream machine learning step.

[0174] By employing the machine learning data exchange method provided in this embodiment of the invention, the global control storage server can determine the target storage strategy based on the data exchange volume and topology information of the data exchange window where the target dataset is located. Then, according to the target storage strategy, the target dataset can be stored in the storage area corresponding to the unified multi-level storage server, thus realizing the orderly data exchange between various machine learning steps in the machine learning pipeline and effectively improving the efficiency of data exchange in the distributed machine learning pipeline. Furthermore, by employing the machine learning data exchange method provided in this embodiment of the invention, the front-end server node does not need to remain running after completing the task, thereby improving the stability of data exchange.

[0175] Optionally, before obtaining the topological relationship information between each machine learning step in the machine learning pipeline, the following step A1 is also included:

[0176] Step A1: Obtain the input-output relationship information between each machine learning step in the machine learning pipeline;

[0177] The step of obtaining the input-output relationship information between each machine learning step in the machine learning pipeline includes: determining the topological relationship information between each machine learning step based on the input-output relationship information.

[0178] Optionally, the system may also include a pipeline scheduling server;

[0179] The acquisition of topological relationship information between various machine learning steps in the machine learning pipeline includes:

[0180] The pipeline scheduling server obtains the topological relationship information between the various machine learning steps; wherein the topological relationship information is determined by the pipeline scheduling server based on the input-output relationship information between the various machine learning steps in the machine learning pipeline.

[0181] Optionally, before determining the target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topology information, the method further includes:

[0182] Based on the aforementioned topological relationship information, the data exchange window is determined using the following formula:

[0183]

[0184] Where n is the number of data exchange windows in the machine learning pipeline, W n W is the nth data exchange window in the machine learning pipeline. n-i Let C0 be the n-th data exchange window in the machine learning pipeline, G0 be the set of vertices with an out-degree of 0 in G0, and G0 be the topological relationship information of each dataset in the machine learning pipeline. i Let G0 be the set of datasets with an out-degree of 0 before the i-th topology pruning. i-1 Let G0 be a set of datasets with an out-degree of 0 before the (i-1)th topology pruning. Topology pruning is defined as deleting datasets with an out-degree of 0 based on their topological relationships. The out-degree of a dataset is the number of times it is used as input to downstream machine learning steps, as determined by the topological relationships. If a dataset is not used as input to any downstream machine learning step, then its out-degree is 0.

[0185] Optionally, the unified multi-level storage server includes a first storage area, a second storage area, and a third storage area; the storage space of the first storage area is smaller than the storage space of the second storage area, and the storage space of the second storage area is smaller than the storage space of the third storage area.

[0186] The step of determining the target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topology information includes:

[0187] Based on the data exchange window where the target dataset is located, and according to the proportion of data exchange volume of each dataset in the data exchange window, the remaining storage space of the first storage area is allocated to the target dataset; if the first storage area cannot completely store the target dataset, the storage space of the second storage area is allocated to the remaining data of the target dataset; if neither the first storage area nor the second storage area can completely store the target dataset, the storage space of the third storage area is allocated to the remaining data of the target dataset, and the proportion of storage areas allocated to the target dataset is used as the target storage strategy.

[0188] Optionally, the step of allocating the remaining storage space of the first storage area to the target dataset according to the proportion of data exchange volume of each dataset in the data exchange window where the target dataset is located includes:

[0189] The remaining storage space in the first storage area is allocated to the target dataset using the following formula, based on the proportion of data exchange volume of each dataset within the data exchange window where the target dataset is located:

[0190]

[0191] in, i-1 d is the set of datasets in the data exchange window where the target dataset is located that have not been allocated storage space in the first storage area. j For datasets without allocated storage space in this data exchange window, C i This is the collection of datasets for which storage space in the first storage region has been allocated within the data exchange window. For dataset d j The maximum remaining storage space in the allocated first storage region, λ out (d j ) for dataset d j The number of times it is used for data exchange, λ out (d α ) for dataset d α The number of times it is used for data exchange, E(d) j ) for dataset d j The size of the data, d α For set C i-1 One of the datasets, E(d) α ) for dataset d α The size of the data, d βFor set C i One of the datasets, Q real (d β ) is for dataset d β The allocated storage space for the first storage region, Q total This refers to the storage space in the first storage area.

[0192] Optionally, before allocating the remaining storage space of the first storage area to the target dataset according to the proportion of data exchange volume of each dataset in the data exchange window where the target dataset is located, the following steps B1-B2 are further included:

[0193] Step B1: Obtain the data volume of each historical exchange dataset in the machine learning pipeline;

[0194] Step B2: Based on the data volume of the historical exchange dataset, calculate the data exchange volume of each dataset in the data exchange window where the target dataset is located.

[0195] This invention addresses the instability of data exchange caused by data volume fluctuations by unifying the massive storage resources of a remote distributed file system in a multi-level storage server. Furthermore, during the execution of the distributed machine learning pipeline, dynamic storage strategies are allocated through data exchange windows and data volume evaluation to achieve global optimization of data exchange in the distributed machine learning pipeline, thereby improving the stability and execution efficiency of data exchange and enhancing the performance of machine learning.

[0196] Corresponding to the data exchange method in machine learning described above, this embodiment of the invention also provides a data exchange device for machine learning. The data exchange device for machine learning provided by this embodiment of the invention is described below. Figure 12 As shown, a data exchange device for machine learning is used in a global control storage server within a machine learning data exchange system. The system also includes a unified multi-level storage server and multiple processing servers, such as... Figure 12 As shown, the device includes:

[0197] The topology acquisition module 1201 is used to acquire the topology relationship information between each machine learning step in the machine learning pipeline.

[0198] The storage strategy determination module 1202 is used to receive a storage strategy request for the target dataset sent by the upstream processing server, and determine the target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topology information; wherein, the target dataset is the dataset output by the target machine learning step obtained after the upstream processing server executes the target machine learning pipeline; the data exchange window is predetermined by the global control storage server based on the topology information, and each data exchange window includes at least one dataset exchanged between machine learning steps; the upstream processing server is the processing server corresponding to the target learning step;

[0199] The storage policy sending module 1203 is used to send the target storage policy to the upstream processing server, so that the upstream processing server stores the target dataset to the storage area corresponding to the unified multi-level storage server according to the received target storage policy.

[0200] The distribution information sending module 1204 is used to send the storage distribution information of the target dataset to the downstream processing server after the downstream machine learning step of the target machine learning step is triggered, so that: the downstream processing server obtains the target dataset from the unified multi-level storage server according to the storage distribution information and executes the operation of writing the target dataset by the downstream machine learning step; wherein, the downstream processing server is the processing server corresponding to the downstream machine learning step.

[0201] Using the machine learning data exchange device provided in this embodiment of the invention, the global control storage server can determine the target storage strategy based on the data exchange volume and topology information of the data exchange window where the target dataset is located. Then, according to the target storage strategy, the target dataset can be stored in the storage area corresponding to the unified multi-level storage server, realizing the orderly data exchange between various machine learning steps in the machine learning pipeline, effectively improving the efficiency of data exchange in the distributed machine learning pipeline. Furthermore, using the machine learning data exchange device provided in this embodiment of the invention, the front-end server node does not need to remain running after completing the task, thus improving the stability of data exchange.

[0202] Optionally, the device further includes:

[0203] The input-output relationship module (not shown in the accompanying drawings) is used to obtain the input-output relationship information between each machine learning step in the machine learning pipeline.

[0204] The topology relationship acquisition module is specifically used to determine the topology relationship information between each machine learning step based on the input-output relationship information.

[0205] Optionally, the system may also include a pipeline scheduling server;

[0206] The topology relationship acquisition module 1201 is specifically used to acquire the topology relationship information between the various machine learning steps from the pipeline scheduling server; wherein, the topology relationship information is determined by the pipeline scheduling server based on the input-output relationship information between the various machine learning steps in the machine learning pipeline.

[0207] Optionally, the device further includes:

[0208] The data exchange window determination module (not shown in the accompanying drawings) is used to determine the data exchange window based on the topology information using the following formula:

[0209]

[0210] Where n is the number of data exchange windows in the machine learning pipeline, W n W is the nth data exchange window in the machine learning pipeline. n-i Let C0 be the n-th data exchange window in the machine learning pipeline, G0 be the set of vertices with an out-degree of 0 in G0, and G0 be the topological relationship information of each dataset in the machine learning pipeline. i Let G0 be the set of datasets with an out-degree of 0 before the i-th topology pruning. i-1 Let G0 be a set of datasets with an out-degree of 0 before the (i-1)th topology pruning. Topology pruning is defined as deleting datasets with an out-degree of 0 based on their topological relationships. The out-degree of a dataset is the number of times it is used as input to downstream machine learning steps, as determined by the topological relationships. If a dataset is not used as input to any downstream machine learning step, then its out-degree is 0.

[0211] Optionally, the unified multi-level storage server includes a first storage area, a second storage area, and a third storage area; the storage space of the first storage area is smaller than the storage space of the second storage area, and the storage space of the second storage area is smaller than the storage space of the third storage area.

[0212] The storage strategy determination module 1202 is specifically used to allocate the remaining storage space of the first storage area to the target dataset according to the data exchange window where the target dataset is located and the proportion of data exchange volume of each dataset in the data exchange window; if the first storage area cannot completely store the target dataset, allocate the storage space of the second storage area to the remaining data of the target dataset; if neither the first storage area nor the second storage area can completely store the target dataset, allocate the storage space of the third storage area to the remaining data of the target dataset, and use the proportion of storage area allocated to the target dataset as the target storage strategy.

[0213] Optionally, the storage strategy determination module 1202 is specifically used to allocate the remaining storage space of the first storage area to the target dataset according to the proportion of data exchange volume of each dataset in the data exchange window where the target dataset is located, using the following formula:

[0214]

[0215] in, i-1 d is the set of datasets in the data exchange window where the target dataset is located that have not been allocated storage space in the first storage area. j For datasets without allocated storage space in this data exchange window, C i This is the collection of datasets for which storage space in the first storage region has been allocated within the data exchange window. For dataset d j The maximum remaining storage space in the allocated first storage region, λ out (d j ) for dataset d j The number of times it is used for data exchange, λ out (d α ) for dataset d α The number of times it is used for data exchange, E(d) j ) for dataset d j The size of the data, d α For set C i-1 One of the datasets, E(d) α ) for dataset d α The size of the data, d β For set C i One of the datasets, Q real (d β ) is for dataset d β The allocated storage space for the first storage region, Q total This refers to the storage space in the first storage area.

[0216] Optionally, the device further includes:

[0217] The data volume acquisition module (not shown in the accompanying drawings) is used to acquire the data volume of each historical exchange dataset of the machine learning pipeline; based on the data volume of the historical exchange dataset, it calculates the data exchange volume of each dataset in the data exchange window where the target dataset is located.

[0218] This invention addresses the instability of data exchange caused by data volume fluctuations by unifying the massive storage resources of a remote distributed file system in a multi-level storage server. Furthermore, during the execution of the distributed machine learning pipeline, dynamic storage strategies are allocated through data exchange windows and data volume evaluation to achieve global optimization of data exchange in the distributed machine learning pipeline, thereby improving the stability and execution efficiency of data exchange and enhancing the performance of machine learning.

[0219] This invention also provides an electronic device, such as... Figure 13 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304. The processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0220] Memory 1303 is used to store computer programs;

[0221] When the processor 1301 executes the program stored in the memory 1303, it implements the above-mentioned data exchange method steps in machine learning.

[0222] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0223] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0224] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0225] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0226] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described methods for exchanging data in machine learning.

[0227] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the data exchange methods in machine learning described above.

[0228] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0229] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0230] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of systems, devices, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0231] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data exchange system for machine learning, characterized in that, include: The system comprises a global control storage server, a unified multi-level storage server, and multiple processing servers; each processing server corresponds to a machine learning step in the machine learning pipeline. An upstream processing server is used to execute the target machine learning step in the machine learning pipeline, obtain the target dataset output by the target machine learning step, and send a storage policy request for the target dataset to the global control storage server; wherein, the upstream processing server is the processing server corresponding to the target learning step; The global control storage server is used to acquire the topological relationship information between each machine learning step in the machine learning pipeline, and after receiving the storage strategy request, determine the target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topological relationship information, and send the target storage strategy to the upstream processing server; wherein, the data exchange window is predetermined by the global control storage server based on the topological relationship information, and each data exchange window includes at least one dataset exchanged between machine learning steps; The upstream processing server is also configured to store the target dataset to the storage area corresponding to the unified multi-level storage server according to the received target storage strategy. A downstream processing server is configured to, after the downstream machine learning step of the target machine learning step is triggered, obtain the storage distribution information of the target dataset from the global control storage server, obtain the target dataset from the unified multi-level storage server according to the storage distribution information, and execute the operation of writing the target dataset by the downstream machine learning step; wherein, the downstream processing server is the processing server corresponding to the downstream machine learning step.

2. The system according to claim 1, characterized in that, The global control storage server is specifically used to acquire the input-output relationship information between each machine learning step in the machine learning pipeline, and to determine the topological relationship information between each machine learning step based on the input-output relationship information.

3. The system according to claim 1, characterized in that, The system also includes a pipeline scheduling server; The pipeline scheduling server is used to determine the topological relationship information between the machine learning steps in the machine learning pipeline based on the input-output relationship information between the machine learning steps in the machine learning pipeline, send the topological relationship information to the global control storage server, and trigger the upstream processing server to execute the target machine learning step in the machine learning pipeline based on the topological relationship information. The global control storage server is specifically used to receive the topology information sent by the pipeline scheduling server; The pipeline scheduling server is also used to trigger the downstream processing server to execute the downstream machine learning step after the target machine learning step is completed.

4. The system according to claim 1, characterized in that, The global control storage server is also used to determine the data exchange window based on the topology information using the following formula: Where n is the number of data exchange windows in the machine learning pipeline, W n W is the nth data exchange window in the machine learning pipeline. n-i Let C0 be the n-th data exchange window in the machine learning pipeline, G0 be the set of vertices with an out-degree of 0 in G0, and G0 be the topological relationship information of each dataset in the machine learning pipeline. i Let C be the set of datasets with an out-degree of 0 before the i-th topology pruning of G0. i-1 Let G0 be a set of datasets with an out-degree of 0 before the (i-1)th topology pruning. Topology pruning is defined as deleting datasets with an out-degree of 0 based on their topological relationships. The out-degree of a dataset is the number of times it is used as input to downstream machine learning steps, as determined by the topological relationships. If a dataset is not used as input to any downstream machine learning step, then its out-degree is 0.

5. The system according to claim 4, characterized in that, The unified multi-level storage server includes a first storage area, a second storage area, and a third storage area; the storage space of the first storage area is smaller than the storage space of the second storage area, and the storage space of the second storage area is smaller than the storage space of the third storage area; The global control storage server is specifically used to allocate the remaining storage space of the first storage area to the target dataset according to the data exchange window where the target dataset is located and the proportion of data exchange volume of each dataset in the data exchange window. If the first storage area fails to fully store the target dataset, the storage space of the second storage area is allocated to the remaining data of the target dataset. If neither the first storage area nor the second storage area can fully store the target dataset, the storage space of the third storage area is allocated to the remaining data of the target dataset. The proportion of storage areas allocated to the target dataset is used as the target storage strategy.

6. The system according to claim 5, characterized in that, The global control storage server is specifically used to allocate the remaining storage space of the first storage area to the target dataset according to the proportion of data exchange volume of each dataset in the data exchange window where the target dataset is located, using the following formula: Among them, C i-1 d is the set of datasets in the data exchange window where the target dataset is located that have not been allocated storage space in the first storage area. j For datasets without allocated storage space in this data exchange window, C i This is the collection of datasets for which storage space in the first storage region has been allocated within the data exchange window. For dataset d j The maximum remaining storage space in the allocated first storage region, λ out (d j ) for dataset d j The number of times it was used for data exchange, λ out (d α ) for dataset d α The number of times it is used for data exchange, E(d) j ) for dataset d j The size of the data, d α For set C i-1 One of the datasets, E(d) α ) for dataset d α The size of the data, d β For set C i One of the datasets, Q real (d β ) is for dataset d β The allocated storage space for the first storage region, Q total This refers to the storage space in the first storage area.

7. The system according to claim 4, characterized in that, The global control storage server is also used to obtain the data volume of each historical exchange dataset of the machine learning pipeline; and based on the data volume of the historical exchange dataset, to calculate the data exchange volume of each dataset in the data exchange window where the target dataset is located.

8. A method for exchanging data in machine learning, characterized in that, A global control storage server is used in a data exchange system for machine learning, the system further including a unified multi-level storage server and multiple processing servers, the method comprising: Obtain the topological relationship information between each machine learning step in the machine learning pipeline; The system receives a storage strategy request for the target dataset from an upstream processing server and determines a target storage strategy based on the data exchange volume of the data exchange window containing the target dataset and the topology information. The target dataset is the dataset output by the target machine learning step after the upstream processing server executes the target machine learning pipeline. The data exchange window is predetermined by the global control storage server based on the topology information, and each data exchange window includes at least one dataset exchanged between machine learning steps. The upstream processing server is the processing server corresponding to the target learning step. The target storage policy is sent to the upstream processing server so that the upstream processing server stores the target dataset in the storage area corresponding to the unified multi-level storage server according to the received target storage policy. After the downstream machine learning step of the target machine learning step is triggered, the storage distribution information of the target dataset is sent to the downstream processing server so that: the downstream processing server obtains the target dataset from the unified multi-level storage server according to the storage distribution information and executes the operation of writing the target dataset by the downstream machine learning step; wherein, the downstream processing server is the processing server corresponding to the downstream machine learning step.

9. The method according to claim 8, characterized in that, The system also includes a pipeline scheduling server; The acquisition of topological relationship information between various machine learning steps in the machine learning pipeline includes: The pipeline scheduling server obtains the topological relationship information between the various machine learning steps; wherein the topological relationship information is determined by the pipeline scheduling server based on the input-output relationship information between the various machine learning steps in the machine learning pipeline.

10. A data exchange device for machine learning, characterized in that, A global control storage server for a data exchange system used in machine learning, the system further including a unified multi-level storage server and multiple processing servers, the device comprising: The topology acquisition module is used to acquire topology relationship information between various machine learning steps in the machine learning pipeline; A storage strategy determination module is used to receive a storage strategy request for the target dataset sent by an upstream processing server, and determine a target storage strategy based on the data exchange volume of the data exchange window where the target dataset is located and the topology information; wherein, the target dataset is the dataset output by the target machine learning step obtained after the upstream processing server executes the target machine learning pipeline; the data exchange window is predetermined by the global control storage server based on the topology information, and each data exchange window includes at least one dataset exchanged between machine learning steps; the upstream processing server is the processing server corresponding to the target learning step; A storage policy sending module is used to send the target storage policy to the upstream processing server, so that the upstream processing server stores the target dataset in the storage area corresponding to the unified multi-level storage server according to the received target storage policy. The distribution information sending module is used to send the storage distribution information of the target dataset to the downstream processing server after the downstream machine learning step of the target machine learning step is triggered, so that: the downstream processing server obtains the target dataset from the unified multi-level storage server according to the storage distribution information and executes the operation of writing the target dataset by the downstream machine learning step; wherein, the downstream processing server is the processing server corresponding to the downstream machine learning step.

Images