Model training methods, platforms and electronic devices

Abstract

The present application discloses a model training method, platform and electronic device, which converts the data to be trained into the first data corresponding to the first programming language through the first cluster, and stores the first data in the preset storage space; obtains the first data in the preset storage space through the second cluster; converts the first data into the second data corresponding to the second programming language through the second cluster, and performs distributed model training according to the second data. Since in the scheme of the present application, the first cluster processes the data according to its corresponding first programming language, and the second cluster performs model training according to its corresponding second programming language, both clusters are combined using their applicable programming languages, which can ensure the operating efficiency of each cluster, and there is no need to limit the two combined clusters to support the same data format, thereby improving the efficiency of neural network model training while performing deep neural network model training on the basis of satisfying large data sets.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a model training method, platform and electronic device. Background Art

[0002] With the rise of artificial intelligence, deep learning has broad application prospects in the field of artificial intelligence. For example, system identification, pattern recognition, intelligent control, etc. are inseparable from deep learning. In the process of deep learning, the model training is achieved by continuously updating the model by derivation using training data. The larger the training data set, the better the performance of the trained model. Therefore, deep neural network models trained on the basis of large data sets are particularly important.

[0003] In the existing technology, clusters that support the same data format are usually combined to achieve deep neural network model training based on large data sets. However, if the clusters are developed based on different languages, the operation efficiency of one or more clusters will be low, which will ultimately lead to low efficiency of model training. Summary of the invention

[0004] The main purpose of this application is to provide a model training method, platform and electronic device, and to provide a model training solution, which aims to solve the technical problem of low model training efficiency when combining different clusters to train neural network models based on large data sets in the prior art.

[0005] To achieve the above object, the present application provides a model training method, which is applied to a model training platform. The model training platform is provided with a first cluster and a second cluster. The first cluster corresponds to a first programming language, and the second cluster corresponds to a second programming language. The model training method includes:

[0006] Converting the data to be trained into first data corresponding to a first programming language through a first cluster, and storing the first data into a preset storage space, wherein the first data is in a first format;

[0007] Acquiring the first data in a preset storage space through the second cluster;

[0008] The first data is converted into second data corresponding to a second programming language through a second cluster, and distributed model training is performed based on the second data, where the second data is in a second format.

[0009] Optionally, converting the to-be-trained data into first data corresponding to the first programming language by using the first cluster includes:

[0010] Dividing the data to be trained into M training data by the first cluster, where M is an integer greater than 1;

[0011] Through the first cluster, a portion of sub-training data corresponding to each of M partitions is determined in the M sub-training data, and the M partitions are the partitions in the first cluster;

[0012] Through the first cluster, format conversion is performed on the corresponding sub-training data in each partition to obtain M pieces of first sub-data, where the first sub-data is in a first format and the first data includes the M pieces of first sub-data.

[0013] Optionally, by using the first cluster, format conversion is performed on the corresponding sub-training data in each partition to obtain M pieces of first sub-data, including:

[0014] For any partition among the M partitions, convert the sub-training data corresponding to the partition into data in a two-dimensional array format in the partition through the first cluster;

[0015] The data in the two-dimensional data format is converted into first sub-data in a one-dimensional array format through the first cluster, and the first format is a one-dimensional array format.

[0016] Optionally, storing the first data in a preset storage space includes:

[0017] Determine, by using the first cluster, a sub-storage space corresponding to each piece of first sub-data, wherein the preset storage space includes sub-storage spaces corresponding to M pieces of first sub-data;

[0018] Each copy of the first sub-data is stored in a corresponding sub-storage space through the first cluster.

[0019] Optionally, the default storage space is off-heap memory.

[0020] Optionally, the preset storage space includes M sub-storage spaces, and N model training processes are set in the second cluster, where N is an integer greater than or equal to 2; and obtaining the first data in the preset storage space through the second cluster includes:

[0021] Determine, by the second cluster, at least one sub-storage space corresponding to each model training process in the M sub-storage spaces;

[0022] The first sub-data in at least one sub-storage space corresponding to each model training process in the second cluster is obtained, and the first data includes the first sub-data in at least M sub-storage spaces.

[0023] Optionally, converting the first data into second data corresponding to the second programming language by the second cluster includes:

[0024] For any one of the N model training processes, merging first sub-data in at least one sub-storage space corresponding to the model training process to obtain merged sub-data;

[0025] The format of the merged sub-data is converted through the second cluster to obtain second sub-data corresponding to the model training process, the format of the second sub-data is a second format, and the second data includes second sub-data corresponding to N model training processes.

[0026] Optionally, performing distributed model training according to the second data includes:

[0027] Determine the system parameters and model parameters corresponding to each model training process through the second cluster;

[0028] For any one model training process among the N model training processes, distributed model training is performed on the second sub-data corresponding to the model training process according to the system parameters and model parameters corresponding to the model training process.

[0029] Optionally, before obtaining the first data in the preset storage space through the second cluster, the method further includes:

[0030] A calling instruction is sent to the second cluster through a preset program, where the calling instruction is used to call the second cluster to perform distributed model training, and the preset program is a program implemented in the first programming language.

[0031] Optionally, the first cluster is a Spark cluster, and the first programming language is Scala or JAVA;

[0032] The second cluster is the Pytorch cluster, and the second programming language is Python.

[0033] The present application also provides a model training platform, in which a first cluster and a second cluster are provided, the first cluster corresponds to a first programming language, and the second cluster corresponds to a second programming language, the model training platform comprises: a data conversion module, for converting the data to be trained into first data corresponding to the first programming language through the first cluster, and storing the first data into a preset storage space, the first data being in a first format;

[0034] An acquisition module, used for acquiring the first data in a preset storage space through the second cluster;

[0035] The processing module is used to convert the first data into second data corresponding to the second programming language through the second cluster, and perform distributed model training according to the second data, where the second data is in a second format.

[0036] The present application also provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the model training method provided in the first aspect when executed by the processor.

[0037] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method provided in the first aspect are implemented.

[0038] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the model training method provided in the first aspect.

[0039] Compared with the current method of combining clusters that support the same data format to implement deep neural network model training based on a large data set, the present application converts the data to be trained into first data corresponding to a first programming language through a first cluster, and stores the first data in a preset storage space, and the first data is in a first format; obtains the first data in the preset storage space through a second cluster; converts the first data into second data corresponding to a second programming language through a second cluster, and performs distributed model training based on the second data, and the second data is in a second format. Because in the scheme of the present application, the first cluster processes data according to its corresponding first programming language, and the second cluster performs model training on the data according to its corresponding second programming language, and both clusters are combined based on their applicable programming languages, the operating efficiency of each cluster can be guaranteed, and there is no need to limit the two combined clusters to support the same data format, thereby improving the efficiency of neural network model training while satisfying the large data set to perform deep neural network model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0041] Figure 2 A flowchart of a model training method provided in one embodiment of the present application;

[0042] Figure 3 A flowchart of a model training method provided in another embodiment of the present application;

[0043] Figure 4 A schematic diagram of the principle of a model training method provided in one embodiment of the present application;

[0044] Figure 5 A schematic diagram of the principle of a model training method provided in another embodiment of the present application;

[0045] Figure 6 A schematic diagram of the structure of a model training platform provided in one embodiment of the present application;

[0046] Figure 7 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0047] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0049] With the rise of artificial intelligence, deep learning has broad application prospects in the field of artificial intelligence. For example, system identification, pattern recognition, intelligent control, etc. are inseparable from deep learning. In the process of deep learning, the model training is achieved by continuously updating the model by derivation using training data. The larger the training data set, the better the performance of the trained model. Therefore, deep neural network models trained on the basis of large data sets are particularly important.

[0050] In the prior art, in order to be compatible with the original Hadoop ecosystem, a single neural network engine platform may be difficult to apply to a large number of usage scenarios, so an effective solution is to adopt a hybrid platform design and expand the platform's functions by integrating different frameworks. Usually, clusters that support the same data format are selected to be combined to achieve deep neural network model training based on large data sets. However, if the clusters are developed based on different languages, the operating efficiency of one or more clusters will be low, which will ultimately lead to low efficiency in model training. For example, Spark and Pytorch are combined. As a large-scale distributed computing engine, Spark introduces the actor parallel model, which has great advantages in processing distributed data sets, making it have excellent performance in big data processing. Since Spark has many excellent features, such as excellent job debugging strategies, simple and easy-to-use interfaces, comprehensive functions, and integrated SQL semantics, streaming computing, machine learning, graph models and other functions, Spark is still a very important part of the Hadoop ecosystem so far.

[0051] Pytorch is an excellent machine learning and deep learning library developed based on Python. It uses a flexible dynamic graph mechanism, making it easy to use and debug. At present, the main functions of Pytorch are relatively mature and are increasingly favored by academia and industry. Among them, it has excellent support for neural networks, which makes it widely used in neural network model training.

[0052] In the prior art, Spark clusters are combined with Pytorch clusters, and the Spark clusters are used to implement distributed data processing, and the Pytorch clusters are used to complete the training of neural network models, so as to realize the training of neural network models based on large data sets. However, since the above combination scheme combines the Python version of the Spark cluster with the Pytorch cluster, and the Spark cluster is actually developed based on the Scala language, the Python version of the Spark cluster is less efficient at runtime, which ultimately leads to low efficiency in training neural network models based on the above combination scheme. As the amount and complexity of data continue to grow, it is difficult to meet the needs of high-performance computing.

[0053] In view of this, the present application provides a model training method, platform and electronic device, in which a first cluster developed in a first programming language is combined with a second cluster developed in a second programming language, the first cluster processes data according to its corresponding first programming language, and the second cluster performs model training on the data according to its corresponding second programming language. Both clusters are combined using their applicable programming languages, which can ensure the operating efficiency of each cluster, and there is no need to restrict the two combined clusters to support the same data format, thereby improving the efficiency of neural network model training while satisfying the requirements of large data sets for deep neural network model training.

[0054] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 As shown, the scenario includes: a model training platform 100 and a preset storage space 104. Wherein, a first cluster 101, a client (driver) 102 and a second cluster 103 are deployed on the model training platform 100.

[0055] In the embodiment of the present application, the model training platform 100 can be any electronic device configured with a processor and a memory, or a collection of electronic devices, such as a server.

[0056] It should be noted that, for ease of understanding, the embodiment of the present application takes the first cluster 101 as a Spark cluster and the second cluster 103 as a Pytorch cluster as an example for explanation. In actual applications, the above clusters can be other types of clusters, and the embodiment of the present application does not make specific limitations.

[0057] In combination with the above, the first cluster 101 is applicable to a first programming language, and the second cluster 103 is applicable to a second programming language, wherein the first programming language may be Scala or JAVA, and the second programming language may be Python.

[0058] Optionally, the preset storage space 104 can be any storage space with data storage function, such as: the local storage space of the model training platform 100, the cloud and the off-heap memory of the cluster, etc., which is not specifically limited in the embodiments of the present application.

[0059] In actual applications, the first cluster 101 is the active caller of the program. The program can be started and data can be loaded through the first cluster 101, and the distributed model can be trained through the second cluster 103. The client (driver) 102 controls the startup and shutdown of the first cluster 101 and the second cluster 103.

[0060] During the model training process, the first cluster 101 is started through the client (driver) 102 , and the first cluster 101 is controlled to load the data to be trained, and the loaded data to be trained is stored in the preset storage space 104 .

[0061] Furthermore, the second cluster 103 has an rpc server process written in Scala language, which is used to receive commands sent through the client 102. When the second cluster 103 receives the rpc command sent through the client 102, the second cluster 103 starts and initializes the corresponding model training process in the second cluster 103 according to the rpc command, and then obtains training data from the preset storage space 104. In the model training process, the model is trained according to the training data. When the training is completed, the training results can be obtained from the second cluster 103 through the client 102.

[0062] Since the first cluster 101 processes data according to its corresponding first programming language, and the second cluster 103 performs model training on the data according to its corresponding second programming language, both clusters are combined using their applicable programming languages, which can ensure the operating efficiency of each cluster, and there is no need to restrict the two combined clusters to support the same data format. This can improve the efficiency of neural network model training while meeting the requirements of large data sets for deep neural network model training.

[0063] Figure 2 A flow chart of a model training method provided in an embodiment of the present application. The execution subject of the embodiment of the present application can be the above-mentioned model training platform, in which a first cluster and a second cluster are provided, the first cluster corresponds to a first programming language, and the second cluster corresponds to a second programming language. Figure 2 As shown, the model training method provided in the embodiment of the present application may include the following steps:

[0064] S201. Convert the data to be trained into first data corresponding to a first programming language through a first cluster, and store the first data into a preset storage space.

[0065] The first data is in a first format.

[0066] In actual applications, the data to be trained may be local training sample data stored by the model training platform, or may be training sample data received or acquired by the model training platform in real time through the network, which is not specifically limited in the embodiments of the present application.

[0067] Secondly, the present application does not impose any specific restrictions on the type of data to be trained and the type of training model, that is, the training method provided in the embodiment of the present application can be used to implement the training of various models. Exemplarily, in an optional implementation, the model can be an image recognition model, and accordingly, the data to be trained in the embodiment of the present application can be image data. Through the method provided in the embodiment of the present application, the training of the image recognition model can be implemented.

[0068] In another optional implementation, the model can be a risk prediction model. Accordingly, the training data in the embodiment of the present application can be the user's asset data, such as deposit time, deposit amount, loan time, loan amount, bad debt record, etc. The method provided in the embodiment of the present application can be used to train the risk prediction model.

[0069] In other optional schemes, the model can also be a credit assessment model. Accordingly, the training data in the embodiment of the present application can be the user's credit data, such as loan data, repayment data, overdue data, etc. The method provided in the embodiment of the present application can realize the training of the credit assessment model.

[0070] It should be understood that the above-mentioned training data types and model types are exemplary and are not intended to be a limitation of the present application.

[0071] It should be noted that the embodiments of the present application do not specifically limit the types of the first cluster and the second cluster. Accordingly, the first programming language and the second programming language are programming languages ​​applicable to the above two clusters, respectively, and the embodiments of the present application do not specifically limit them.

[0072] As an optional solution, the first cluster can be a Spark cluster, the second cluster can be a Pytorch cluster, and accordingly, the first programming language can be Scala or JAVA, and the second programming language can be Python. For ease of understanding, the embodiment of the present application is described by taking the first cluster as a Spark cluster, the second cluster as a Pytorch cluster, the first programming language as Scala, and the second programming language as Python as an example.

[0073] In this step, after the data to be trained is obtained, the data to be trained is converted into first data corresponding to the first programming language through the first cluster, wherein the first data is in a first format. The first format is not specifically limited in the embodiment of the present application, and illustratively, the first format can be a one-dimensional array format, a two-dimensional array format, etc.

[0074] In this step, since the format of the acquired training data is not uniform, it can be converted into the first format before model training to facilitate the subsequent data processing process and improve data processing efficiency.

[0075] Furthermore, the first data in a one-dimensional array format is stored in a preset storage space. On the one hand, the preset storage space can be any storage space with storage function locally on the model training platform. By storing the training data locally on the model training platform, there is no need to transmit the data through a network or other means, thereby ensuring the security of the data.

[0076] On the other hand, the preset storage space is also the storage space of the cloud. Since the data volume of the training samples is large, the training data is stored in the cloud, and then the data to be trained is obtained in real time according to the demand, which can decouple the data storage pressure of the model training platform and ensure the operation performance of the model training platform. In other embodiments, the preset storage space can also be off-heap memory. When using off-heap memory to store data, the data is stored in an external medium through the built-in commonly used adapter. When performing model training, it has the characteristic of zero copy, that is, the data to be trained can be obtained directly from the off-heap memory without copying, thereby improving the data transmission efficiency and further improving the model training efficiency.

[0077] S202: Acquire first data in a preset storage space through a second cluster.

[0078] It should be noted that after the first cluster stores the first data in the preset storage space, in this step, the second cluster directly obtains the first data from the preset storage space, that is, the two share the preset storage space. Since the amount of first data is large, this solution does not require data transmission, thereby improving the data acquisition efficiency of the second cluster and ultimately improving the model training efficiency.

[0079] S203: Convert the first data into second data corresponding to a second programming language through a second cluster, and perform distributed model training based on the second data.

[0080] The second data is in the second format. The embodiment of the present application does not specifically limit the type of the second format. For example, the second cluster is a Pytorch cluster and the second programming language is Python. Correspondingly, the second format can be a data structure Tensor representing a tensor.

[0081] Furthermore, distributed model training is performed based on the second data.

[0082] Optionally, in order to ensure the security of the data, the training data can be encrypted when the Spark cluster stores the training data in the preset storage space. Correspondingly, when the training data is obtained by the second cluster, it needs to be decrypted. The encryption method is not specifically limited in the embodiment of the present application. The above means can meet the needs of privacy-preserving computing and further improve the security of the model training platform.

[0083] In the present application, the data to be trained is converted into first data corresponding to the first programming language by the first cluster, and the first data is stored in a preset storage space, and the first data is in a first format; the first data is obtained in the preset storage space by the second cluster; the first data is converted into second data corresponding to the second programming language by the second cluster, and distributed model training is performed based on the second data, and the second data is in a second format. Because in the scheme of the present application, the first cluster processes data according to its corresponding first programming language, and the second cluster performs model training on the data according to its corresponding second programming language, both clusters are combined using their applicable programming languages, which ensures the operating efficiency of each cluster, and there is no need to limit the two combined clusters to support the same data format, thereby improving the efficiency of neural network model training while performing deep neural network model training on the basis of satisfying large data sets.

[0084] In addition, since the first cluster and the second cluster share the preset storage space, after the first cluster stores the first data in the preset storage space, the second cluster directly obtains the first data from the preset storage space. This solution does not require data transmission, thereby improving the data acquisition efficiency of the second cluster and ultimately improving the model training efficiency.

[0085] Figure 3 A flowchart of a model training method provided for another embodiment of the present application. Based on the above embodiment, this embodiment describes the technical solution of the present application in more detail. The model training method provided in the embodiment of the present application is applied to a model training platform, and a first cluster and a second cluster are provided in the model training platform, the first cluster corresponds to a first programming language, and the second cluster corresponds to a second programming language.

[0086] It should be noted that the embodiment of the present application is still described by taking the first cluster as a Spark cluster, the second cluster as a Pytorch cluster, the first programming language as Scala, and the second programming language as Python as an example.

[0087] In actual applications, the Spark cluster is the active caller of the program, responsible for starting the program, loading data, processing, and executing business logic. The Pytorch cluster is responsible for training distributed models. For easier understanding, please refer to Figure 4 , Figure 4 The schematic diagram of the principle of the model training method provided in one embodiment of the present application. Figure 4 As shown in the figure, the driver (client) of the Spark cluster acts as both the driver of Spark and the driver of the Pytorch cluster. It is used to start, control and shut down the Pytorch cluster, send Pytorch code running commands, broadcast data to the Pytorch cluster, and obtain training results from the Pytorch cluster.

[0088] N model training processes are set in the Pytorch cluster, where N is an integer greater than or equal to 2. The value of N can be configured according to the model training requirements, and this embodiment of the application does not specifically limit it. Figure 4 As shown, the embodiment of the present application takes the value of N as 2 as an example, that is, two model training processes are set in the Pytorch cluster.

[0089] In actual applications, the Pytorch cluster has a constant rpc server process written in Scala on each server, which is used to receive and execute commands sent from the driver. Spark starts the Pytorch cluster by sending rpc commands to each Pytorch cluster through the driver, starting and initializing the process with the corresponding number of processes pre-set.

[0090] like Figure 3 As shown, the model training method provided in the embodiment of the present application may include the following steps:

[0091] S301, dividing the data to be trained into M components of training data through a first cluster.

[0092] S302: Determine, through the first cluster, a portion of sub-training data corresponding to each of M partitions in the M sub-training data, wherein M is an integer greater than 1, and the M partitions are partitions in the first cluster.

[0093] It is understandable that the data to be trained is all the sample data used in model training. In practical applications, before dividing the data to be trained into M sub-training data, it is necessary to construct a resilient distributed dataset (RDD) based on Spark. It is understandable that the number of RDDs can be set according to the size of the data to be trained, and the embodiment of the present application does not make specific limitations.

[0094] Further, after obtaining the data to be trained, the data to be trained is divided into M sub-training data, and stored in M ​​RDD partitions of the Spark cluster respectively. Wherein, M is an integer greater than 1, and the value of M can be set according to actual needs, and the embodiment of the present application does not specifically limit it.

[0095] In one embodiment, the value of M can be a fixed value, for example, the value of M is 5; in another embodiment, the value of M can be determined according to the amount of data to be trained. Specifically, since the capacity of each partition is limited, the data to be trained is evenly divided according to the capacity of each partition to determine the value of M, and then the data to be trained is evenly divided into M partitions.

[0096] In other implementations, the determination may be made based on the total number of partitions in the Spark cluster. For example, if five RDD partitions are deployed in the Spark cluster, the value of M is 5, and the data to be trained is evenly divided into the five RDD partitions.

[0097] The above-mentioned data partitioning scheme provided in the embodiment of the present application can adapt to different model training platforms, thereby meeting the current diverse model training needs.

[0098] For easier understanding, please refer to Figure 4 , Figure 4 The schematic diagram of the principle of the model training method provided in one embodiment of the present application. Figure 4 As shown, the Spark cluster provided in the embodiment of the present application is deployed with partition1 to partition(n)n RDD model partitions, wherein each RDD model partition corresponds to a computing node. After the Spark cluster receives the data to be trained, it divides the data to be trained equally into each RDD model partition, thereby determining a portion of training data corresponding to each of the M partitions in the M portions of training data.

[0099] In this step, by distributing the data to be trained to different model partitions in the Spark cluster, the sub-training data corresponding to each partition is determined, so that the Spark cluster performs distributed computing through different model partitions, thereby improving data processing efficiency.

[0100] S303 . Perform format conversion on the corresponding sub-training data in each partition through the first cluster to obtain M pieces of first sub-data.

[0101] The first sub-data is in a first format, and the first data includes M copies of the first sub-data.

[0102] Specifically, for any one of the M partitions, the sub-training data corresponding to the partition is converted into data in a two-dimensional array format in the partition by the first cluster;

[0103] In practical applications, since the format of the acquired training data is not uniform, it can be converted into a two-dimensional array format before model training to facilitate subsequent data processing and improve data processing efficiency.

[0104] Furthermore, the data in the two-dimensional data format is converted into first sub-data in a one-dimensional array format through the first cluster, and the first format is a one-dimensional array format.

[0105] It should be noted that the scheme of this step is Figure 2 The method and principle in the embodiment are similar, and can be referred to Figure 2 The embodiments shown will not be described in detail here.

[0106] S304: Determine, through the first cluster, a sub-storage space corresponding to each piece of first sub-data.

[0107] S305 . Store each copy of the first sub-data in a corresponding sub-storage space through the first cluster.

[0108] The preset storage space includes sub-storage spaces corresponding to M portions of the first sub-data.

[0109] For easier understanding, please refer to Figure 4 The preset storage space includes M sub-storage spaces. The types of sub-storage spaces are the same as those of the preset storage space. For the types of sub-storage spaces, please refer to Figure 2 Step S202 in the embodiment shown is not described in detail here. Figure 4 Take the off-heap memory as a Vector data storage structure as an example.

[0110] In practical applications, each RDD partition corresponds to a Vector. After obtaining M copies of the first sub-data, the M copies of the first sub-data are stored in the Vector corresponding to each RDD partition.

[0111] In one embodiment, the RDD partitions and Vectors in the Spark cluster may be in a one-to-one correspondence, that is, each RDD partition is provided with a corresponding Vector. For example, when there are M partitions in the Spark cluster, there are M Vectors in total, and each partition stores the sub-training data in the partition in the Vector corresponding to the partition.

[0112] In another embodiment, the RDD partitions in the Spark cluster can have a many-to-one relationship with the Vector, that is, multiple partitions correspond to the same Vector. The embodiment of the present application does not limit the number of Vectors. For example, when the number of Vectors is 3, the training data in the M partitions in the Spark cluster can be evenly divided into the above 3 Vectors. It should be noted that Figure 4 The one-to-one correspondence between RDD partitions and Vectors is used as an example.

[0113] In this solution, since Vector uses off-heap memory to store data, it has the characteristic of zero copy, and comes with a commonly used adapter to store data in an external medium, its transmission rate is relatively high, which can further improve the model training efficiency of this application.

[0114] In some embodiments, after the training data is stored in the storage space, the second cluster can be used for model training. Specifically, N model training processes are set in the second cluster, each process executes the same logic, and performs model training according to the training data, where N is an integer greater than or equal to 2. The model training process is described in detail below in conjunction with specific embodiments.

[0115] S306. Determine, through the second cluster, at least one sub-storage space corresponding to each model training process in the M sub-storage spaces.

[0116] S307. Obtain the first sub-data in at least one sub-storage space corresponding to each model training process in the second cluster.

[0117] The first data includes first sub-data in at least M sub-storage spaces.

[0118] In the embodiment of the present application, there is no specific limitation on the way in which each model training process obtains sub-data. In one embodiment, the sub-data can be obtained by an equal division method, that is, the number of sub-storage spaces where the first sub-data obtained by each model training process is the same, and the first sub-data in the remaining sub-storage space after equal division is obtained by any model training process.

[0119] The following process description takes the Spark cluster partition number M as 5 and the Pytorch cluster model training process number N as 2 as an example.

[0120] Please continue to refer to Figure 4 ,like Figure 4As shown, model training process 1 and model training process 2 respectively obtain the first sub-data in two Vectors, that is, model training process 1 obtains the first sub-data in the Vector corresponding to partition 1 and partition 2, and model training process 2 obtains the first sub-data in the Vector corresponding to partition 4 and partition 5. The first sub-data in the Vector corresponding to partition 3 can be obtained by either model training process 1 or model training process 2. Figure 4 Take the first sub-data in the Vector corresponding to partition 3 obtained by model training process 1 as an example.

[0121] The method for obtaining the first sub-data from the Vector during the model training process is not specifically limited in the embodiments of the present application. Since there is no interface for mutual invocation between the Scala language and the Python language, however, the clusters where the Scala programming language, the C language, and the Python language are located can access the same memory. Therefore, in an optional solution, a C language interface can be set in the second cluster to obtain the first sub-data through the C language interface. It should be noted that the embodiments of the present application can refer to the prior art for the method of using the C language as an interface, and the embodiments of the present application are not specifically limited.

[0122] Specifically, since Vector is a data block with continuous addresses, when the first data is stored in Vector through the Spark cluster, each training data has a different data address and data length. When the first sub-data is obtained through the Pytorch cluster, the corresponding first sub-data is obtained from Vector according to different data addresses and data lengths through the C language interface.

[0123] In this solution, C language is used as the interface between the first cluster and the second cluster, so that the second cluster and the first cluster can share off-heap memory, so that when obtaining training data, there is no need to copy and transfer data, which greatly reduces the data acquisition time and further improves the data acquisition efficiency.

[0124] S308. For any one model training process among the N model training processes, merge the first sub-data in at least one sub-storage space corresponding to the model training process to obtain merged sub-data.

[0125] Optionally, for any model training process, the first sub-data in at least one sub-storage space corresponding to the model training process are merged to obtain merged sub-data, and the merged sub-data are stored in the off-heap memory (Vector). It can be understood that the merged sub-data is all the first sub-data corresponding to each model training process.

[0126] Please continue to refer to Figure 4 ,like Figure 4 As shown, model training process 1 corresponds to the first sub-data in the three Vectors corresponding to partitions 1 to 3 in the Spark cluster. After obtaining the first sub-data in the above three Vectors through model training process 1, the first sub-data in the three Vectors in model training process 1 are merged to obtain a merged sub-data, and the merged sub-data is stored in the off-heap memory.

[0127] Correspondingly, the first sub-data in the two Vectors corresponding to partitions 4 and 5 in the Spark cluster are obtained through model training process 2. Further, the first sub-data in the two Vectors corresponding to model training process 2 are merged to obtain a merged sub-data, and the merged Vector is stored in the off-heap memory. Figure 4 The off-heap memory in is shown as a Vector.

[0128] Through this solution, since the first sub-data in each model training process is merged and stored in the same external memory, when the model is trained through the Pytorch cluster, each model training process can directly call the merged sub-data corresponding to the model training process as a whole, without the need to call them separately during the training process, which can reduce the running time of the training process and improve the efficiency of model training.

[0129] S309: Convert the format of the merged sub-data through the second cluster to obtain second sub-data corresponding to the model training process.

[0130] The format of the second sub-data is the second format, and the second data includes the second sub-data corresponding to N model training processes.

[0131] It is understandable that the data structure applicable to the Scala language is the first format, and the data structure applicable to the Python language is the second format. Since the merged sub-data is still in the first format, the Pytorch cluster developed based on the Python language cannot be directly called. Therefore, it is necessary to convert it to the second format so that the Pytorch cluster can be directly called. The second format is not specifically limited in the embodiment of the present application. Exemplarily, the second format can be the Tensor format representing tensors in the Pytorch cluster.

[0132] Specifically, converting the format of the merged sub-data into the second format specifically includes the following steps:

[0133] First, the merged sub-data in the first format are converted into third data in a third format, wherein the third format is the ndarray format in numpy.

[0134] Further, the third data in the third format is converted into the second data in the second format, that is, the third data in the ndarray format in numpy is converted into the second data in the Tensor format in the Pytorch cluster.

[0135] Through this solution, the data format of the merged sub-data can be converted into a data format suitable for the Pytorch cluster, so that the Pytorch cluster can perform model training based on the converted data.

[0136] S310. Determine system parameters and model parameters corresponding to each model training process through the second cluster.

[0137] S311. For any one model training process among the N model training processes, distributed model training is performed on the second sub-data corresponding to the model training process according to the system parameters and model parameters corresponding to the model training process.

[0138] Specifically, for each model training process, the sub-second data in the Tensor format is subjected to model training using the model training mechanism in the model training process, wherein the model training mechanism may be a distributed model mechanism, specifically, a distributed data parallel mechanism (Distributed Data Parallel, DDP).

[0139] Through this solution, since the DDP mechanism is a process-based distributed model mechanism of the Pytorch cluster, it can transform the local model into a distributed training model with almost no modification, and can reuse a large number of Pytorch models, which can significantly improve the model development efficiency and model operation efficiency.

[0140] For ease of understanding, the following Figure 5 The model training principle of this application is explained.

[0141] Figure 5 This is a schematic diagram of the principle of a model training method provided by another embodiment of the present application. Figure 5 As shown, for each model training process in the Python cluster, the second data in the second format, the model parameters, and the system parameters are input into the DDP model of each model training process.

[0142] Among them, the model parameter is an array used to store user-defined parameters, and the system parameter is an indispensable variable for each process when running the DDP model. Specifically, the system parameters may include parameters such as world_size, rank, master_host, and master_port. Among them, world_size is used to indicate the total number of processes called in the Pytorch cluster, rank indicates the current process id, the initial value of the process id is 0, that is, the process id of process 1 is 0, master_host is used to indicate the server IP where the process responsible for managing the process group in the DDP model is located, and master_port is used to indicate the port where the process responsible for managing the process group in the DDP model is located.

[0143] Furthermore, the DDP model in each model training process uses All-Reduce to train and learn the second sub-data in the process to obtain a training result.

[0144] It can be understood by those skilled in the art that, since the present solution uses the All-Reduce algorithm, the training results in each model training process in the Python cluster are the same, and the training results of any model training process can be determined as the trained model parameters. Since the training results obtained by each process are the same, the results of any process can be obtained as the training results at will, thereby simplifying the training result acquisition process. Exemplarily, the training results of the process with process ID 0 can be determined as the trained model parameters.

[0145] In practical applications, after training is completed, the trained model parameters need to be sent to the client (driver) to output the trained model parameters. Since the trained model parameters are in the second format, and the client (driver) is developed based on the Scala language or the JAVA language, it is necessary to convert the trained model parameters in the second format into the first format applicable to the Scala language.

[0146] Specifically, the trained model parameters in the second format (Tensor format) are first converted to the ndarray format, and then the trained model parameters in the ndarray format are converted to the first format.

[0147] Furthermore, in one implementation, after obtaining the model parameters in the first format, the model parameters in the first format may be output to a client (driver).

[0148] In another implementation, the model parameters in the first format may also be stored in a preset storage space.

[0149] For the types of preset storage space, refer to Figure 2 The embodiments shown are not described here in detail. Exemplarily, after the model parameters in the second format are converted into the model parameters in the first format, the model parameters in the first format can be stored in the off-heap memory, and when the data pull command sent by the client (driver) is received, the model parameters are directly obtained from the off-heap memory through the second cluster. By converting the data into the first format, the client (driver) can directly obtain the trained model parameters to achieve the output of the model parameters. In addition, since the trained model parameters are stored in the off-heap memory, the off-heap memory has the characteristic of zero copy. When obtaining the model parameters, there is no need for data transmission, which can improve the efficiency of data acquisition.

[0150] In practical applications, the method of the embodiment of the present application can be used to implement the training of various models. In an optional implementation, the model can be an image recognition model, and accordingly, the training data in the embodiment of the present application can be a face image. Through the above method, the training of the image recognition model can be implemented, and the obtained training model can be used for face recognition. Specifically, when using the training model, the face image data can be input into the image recognition model to obtain the corresponding pedestrian ID, which can assist in pedestrian recognition, reduce the time-consuming and labor-intensive problem of manual recognition, and effectively improve the efficiency and accuracy of recognition.

[0151] In another optional implementation, the model can be a risk prediction model. Accordingly, the training samples in the embodiment of the present application can be the user's asset data such as deposit time, deposit amount, loan time, loan amount, bad debt record, etc. Through the above method, the risk prediction model can be trained, and the obtained global model can be used for risk prediction. Optionally, when using the model, the asset data of the user to be predicted can be input into the model to obtain the corresponding risk prediction results, such as whether the probability of overdue repayment is high or low, and the results can be displayed to the staff, or the loan restrictions on the user can be adjusted according to the results, thereby effectively helping the staff to predict the corresponding risk level of the user, reducing the manual burden, and improving the efficiency and accuracy of the prediction.

[0152] In addition, it should be noted that the execution order of the steps in each embodiment of the present application is not limited to the order specified by the above serial numbers. Those skilled in the art can make any configuration according to specific application requirements and design requirements, which will not be repeated here.

[0153] Figure 6 This is a schematic diagram of the structure of the model training platform provided in one embodiment of the present application. Figure 6 As shown, the model training platform 600 is provided with a first cluster and a second cluster, the first cluster corresponds to a first programming language, and the second cluster corresponds to a second programming language, and the model training platform includes:

[0154] The data conversion module 601 is used to convert the training data into first data corresponding to the first programming language through the first cluster, and store the first data in a preset storage space, where the first data is in a first format; the acquisition module 602 is used to acquire the first data in the preset storage space through the second cluster;

[0155] The processing module 603 is used to convert the first data into second data corresponding to the second programming language through the second cluster, and perform distributed model training according to the second data, where the second data is in a second format.

[0156] Optionally, the data conversion module 601 is specifically used for:

[0157] Dividing the data to be trained into M training data by the first cluster, where M is an integer greater than 1;

[0158] Through the first cluster, a portion of sub-training data corresponding to each of M partitions is determined in the M sub-training data, and the M partitions are the partitions in the first cluster;

[0159] Through the first cluster, format conversion is performed on the corresponding sub-training data in each partition to obtain M pieces of first sub-data, where the first sub-data is in a first format and the first data includes the M pieces of first sub-data.

[0160] Optionally, a data conversion module 601 is specifically configured to convert, for any one of the M partitions, the sub-training data corresponding to the partition into data in a two-dimensional array format in the partition through the first cluster;

[0161] The data in the two-dimensional data format is converted into first sub-data in a one-dimensional array format through the first cluster, and the first format is a one-dimensional array format.

[0162] Optionally, the data conversion module 601 is specifically configured to determine, through the first cluster, a sub-storage space corresponding to each first sub-data, wherein the preset storage space includes the sub-storage spaces corresponding to the M first sub-data;

[0163] Each copy of the first sub-data is stored in a corresponding sub-storage space through the first cluster.

[0164] Optionally, the default storage space is off-heap memory.

[0165] Optionally, the preset storage space includes M sub-storage spaces, and N model training processes are set in the second cluster, where N is an integer greater than or equal to 2.

[0166] An acquisition module 602 is specifically configured to determine, through the second cluster, at least one sub-storage space corresponding to each model training process in the M sub-storage spaces;

[0167] The first sub-data in at least one sub-storage space corresponding to each model training process in the second cluster is obtained, and the first data includes the first sub-data in at least M sub-storage spaces.

[0168] Optionally, the processing module 603 is specifically used to merge the first sub-data in at least one sub-storage space corresponding to any one of the N model training processes to obtain merged sub-data;

[0169] The format of the merged sub-data is converted through the second cluster to obtain second sub-data corresponding to the model training process, the format of the second sub-data is a second format, and the second data includes second sub-data corresponding to N model training processes.

[0170] Optionally, the processing module 603 is specifically configured to determine the system parameters and model parameters corresponding to each model training process through the second cluster;

[0171] For any one model training process among the N model training processes, distributed model training is performed on the second sub-data corresponding to the model training process according to the system parameters and model parameters corresponding to the model training process.

[0172] Optionally, the processing module 603 is further used to send a calling instruction to the second cluster through a preset program, where the calling instruction is used to call the second cluster to perform distributed model training, and the preset program is a program implemented in the first programming language.

[0173] Optionally, the first cluster is a Spark cluster, and the first programming language is Scala or JAVA;

[0174] The second cluster is the Pytorch cluster, and the second programming language is Python.

[0175] It should be noted that the implementation principle and technical effects of the model training platform provided in this embodiment can be found in the aforementioned Figure 2 , Figure 3 The solution provided in any method embodiment will not be repeated here.

[0176] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device may include: a memory 701, a processor 702, and a data processing program stored in the memory 701 and executable on the processor 702. When the data processing program is executed by the processor 702, the steps of the model training method in any of the aforementioned embodiments are implemented.

[0177] Optionally, the memory 701 may be independent or integrated with the processor 702 .

[0178] The implementation principle and technical effects of the electronic device provided in this embodiment can be found in the aforementioned embodiments and will not be described in detail here.

[0179] An embodiment of the present application also provides a computer-readable storage medium, on which a data processing program is stored. When the data processing program is executed by a processor, the steps of the model training method in any of the aforementioned embodiments are implemented.

[0180] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the model training method in the above method embodiment.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0182] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the methods of various embodiments of the present application.

[0183] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0184] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0185] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0186] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0187] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0188] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0189] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0190] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A model training method, characterized in that: Applied to a model training platform, the model training platform is provided with a first cluster and a second cluster, the first cluster corresponds to a first programming language, the second cluster corresponds to a second programming language, and the second cluster is provided with N model training processes, where N is an integer greater than or equal to 2; the method comprises: The data to be trained is converted into first data corresponding to the first programming language by the first cluster, and the first data is stored in a preset storage space, wherein the first data is in a first format, and the preset storage space includes M sub-storage spaces; the data to be trained is image data; Determine, by the second cluster, at least one sub-storage space corresponding to each model training process in the M sub-storage spaces; Acquire, through each model training process in the second cluster, first sub-data in at least one sub-storage space corresponding to each of the two processes, where the first data includes the first sub-data in the M sub-storage spaces; The first data is converted into second data corresponding to the second programming language through the second cluster, and distributed model training is performed based on the second data, where the second data is in a second format.

2. The method according to claim 1, characterized in that: The converting the to-be-trained data into first data corresponding to the first programming language by using the first cluster includes: Dividing the to-be-trained data into M sub-training data by using the first cluster, where M is an integer greater than 1; Determine, through the first cluster, a portion of sub-training data corresponding to each of M partitions in the M sub-training data, wherein the M partitions are partitions in the first cluster; Through the first cluster, format conversion is performed on the corresponding sub-training data in each partition to obtain M pieces of first sub-data, where the first sub-data is in the first format and the first data includes the M pieces of first sub-data.

3. The method according to claim 2, characterized in that The first cluster is used to convert the format of the corresponding sub-training data in each partition to obtain M pieces of first sub-data, including: For any one of the M partitions, converting the sub-training data corresponding to the partition into data in a two-dimensional array format in the partition by using the first cluster; The data in the two-dimensional array format is converted into first sub-data in a one-dimensional array format through the first cluster, and the first format is the one-dimensional array format.

4. The method according to claim 2, characterized in that: The storing the first data into a preset storage space includes: Determine, by using the first cluster, a sub-storage space corresponding to each piece of first sub-data, wherein the preset storage space includes the sub-storage spaces corresponding to the M pieces of first sub-data; Each copy of the first sub-data is stored in a corresponding sub-storage space through the first cluster.

5. The method according to claim 4, characterized in that The preset storage space is off-heap memory.

6. The method according to claim 1, characterized in that The converting the first data into second data corresponding to the second programming language by the second cluster includes: For any one model training process among the N model training processes, merging first sub-data in at least one sub-storage space corresponding to the model training process to obtain merged sub-data; The format of the merged sub-data is converted through the second cluster to obtain second sub-data corresponding to the model training process, the format of the second sub-data is a second format, and the second data includes the second sub-data corresponding to the N model training processes.

7. The method according to claim 6, characterized in that The performing distributed model training according to the second data includes: Determine the system parameters and model parameters corresponding to each model training process through the second cluster; For any one model training process among the N model training processes, distributed model training is performed on the second sub-data corresponding to the model training process according to the system parameters and model parameters corresponding to the model training process.

8. The method according to any one of claims 1 to 7, characterized in that: Also includes: A calling instruction is sent to the second cluster through a preset program, where the calling instruction is used to call the second cluster to perform distributed model training, and the preset program is a program implemented in the first programming language.

9. The method according to any one of claims 1 to 7, characterized in that: The first cluster is a Spark cluster, and the first programming language is Scala or JAVA; The second cluster is a Pytorch cluster, and the second programming language is Python.

10. A model training platform, characterized in that: The model training platform is provided with a first cluster and a second cluster, the first cluster corresponds to a first programming language, the second cluster corresponds to a second programming language, and the second cluster is provided with N model training processes, where N is an integer greater than or equal to 2; The model training platform includes: a data conversion module, configured to convert the data to be trained into first data corresponding to the first programming language through the first cluster, and store the first data into a preset storage space, wherein the first data is in a first format, and the preset storage space includes M sub-storage spaces; the data to be trained is image data; An acquisition module, configured to determine, through the second cluster, at least one sub-storage space corresponding to each model training process in the M sub-storage spaces; Acquire, through each model training process in the second cluster, first sub-data in at least one sub-storage space corresponding to each of the two processes, where the first data includes the first sub-data in the M sub-storage spaces; A processing module is used to convert the first data into second data corresponding to the second programming language through the second cluster, and perform distributed model training according to the second data, wherein the second data is in a second format.

11. An electronic device, characterized in that: The electronic device comprises: A memory, a processor, and a model training program stored in the memory and executable on the processor, wherein the model training program, when executed by the processor, implements the model training method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a model training program, and when the model training program is executed by the processor, it implements the model training method as described in any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the model training method described in any one of claims 1 to 9 is implemented.

Images