Spark cluster-based model training method, device, equipment and storage medium

Abstract

The present invention discloses a model training method based on a Spark cluster, including: obtaining training samples; generating multiple candidate subnetworks through a deep neural subnetwork generator, and based on the training samples, determining the training errors corresponding to each candidate subnetwork through multiple first elastic distributed data sets and each candidate subnetwork in the Spark cluster; determining the optimal subnetwork in each candidate subnetwork based on each training error, and determining the target model based on the optimal subnetwork and the main network. The present invention also discloses a model training device, equipment and storage medium based on a Spark cluster. The present invention realizes model training through elastic distributed data sets in a Spark cluster, and thus does not need to perform model training through a GPU server. It can also realize intelligent training and optimization of different network models according to different training samples, reducing the workload of model training and tuning, and improving the training efficiency of network models.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a Spark cluster-based model training method, device, equipment and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, neural network models (can be referred to as network models for short) have broad application prospects in the fields of system identification, pattern recognition, intelligent control, etc. At the same time, with the rapid development of the information society, the data generated by the Internet is growing rapidly, and distributed computing is usually used to process massive data.

[0003] Spark is a memory-based distributed computing system that has been widely used in the industry for big data processing and traditional machine learning model training. Deep learning models mainly use GPUs for model training. Most companies in traditional industries generally only deploy one Spark cluster and do not configure GPU servers separately, which causes model training to take a lot of time and leads to low efficiency.

[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0005] The main purpose of the present invention is to provide a Spark cluster-based model training method, device, equipment and storage medium, aiming to solve the technical problem of low efficiency of system model training in existing Spark cluster deployments.

[0006] To achieve the above object, the present invention provides a Spark cluster-based model training method, which comprises the following steps:

[0007] Get training samples;

[0008] Generate multiple candidate subnetworks through a deep neural subnetwork generator, and based on the training samples, determine the training error corresponding to each candidate subnetwork through multiple first elastic distributed data sets in a Spark cluster and each candidate subnetwork;

[0009] Based on each of the training errors, an optimal sub-network is determined among each of the candidate sub-networks, and a target model is determined based on the optimal sub-network and the main network.

[0010] In one embodiment, the step of determining the training error corresponding to each candidate subnetwork based on the training sample through the first elastic distributed data set in the Spark cluster and each candidate subnetwork includes:

[0011] Determine target network parameters corresponding to each candidate subnetwork based on the network parameters of each candidate subnetwork and the main network parameters of the main network;

[0012] Based on the training samples, a training error corresponding to each candidate subnetwork is determined through multiple first elastic distributed data sets in a Spark cluster and each target network parameter.

[0013] In one embodiment, the step of determining the training error corresponding to each candidate subnetwork based on the training sample through multiple first elastic distributed data sets in the Spark cluster and each target network parameter includes:

[0014] Allocating each target network parameter to each first elastic distributed data set in the Spark cluster, so that the first elastic distributed data set is trained based on the training sample through the target network parameter;

[0015] The training results corresponding to each first elastic distributed data set are obtained through the second elastic distributed data set, and the training errors corresponding to each candidate sub-network are determined based on the training results.

[0016] In one embodiment, the step of determining the optimal subnetwork among the candidate subnetworks based on the training errors includes:

[0017] Based on the network parameters, obtaining a penalty coefficient corresponding to the number of network layers of each candidate subnetwork;

[0018] Based on each penalty coefficient and training error, the weight score corresponding to each candidate sub-network is determined respectively;

[0019] The candidate subnetwork with the smallest weight score among the candidate subnetworks is taken as the optimal subnetwork.

[0020] In one embodiment, the Spark cluster-based model training method further includes:

[0021] When generating multiple candidate sub-networks through the deep neural sub-network generator, update the search count;

[0022] The step of determining the target model based on the optimal sub-network and the main network comprises:

[0023] Determine whether the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network;

[0024] If the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network, determining whether the number of searches reaches a preset number of searches;

[0025] When the number of searches reaches the preset number of searches, the target model is determined based on the optimal sub-network and the main network.

[0026] In one embodiment, after the step of determining whether the number of searches reaches a preset number of searches, the method further includes:

[0027] When the search times do not reach the preset search times, the network parameters corresponding to the optimal sub-network and the main network parameters of the main network are used as the main network parameters of the main network, and the step of generating multiple candidate sub-networks through the deep neural sub-network generator is returned to be executed.

[0028] In one embodiment, after the step of determining whether the training error corresponding to the optimal sub-network is less than the training error corresponding to the main network, the step further includes:

[0029] If the training error corresponding to the optimal sub-network is greater than or equal to the training error corresponding to the main network, the process returns to the step of generating multiple candidate sub-networks through the deep neural sub-network generator.

[0030] In one embodiment, the step of determining the training error corresponding to each candidate subnetwork based on the training sample through multiple first elastic distributed data sets and each candidate subnetwork in the Spark cluster includes:

[0031] Based on the number of candidate subnetworks, create the first elastic distributed dataset in the Spark cluster;

[0032] Based on the training samples, a training error corresponding to each candidate subnetwork is determined through multiple first elastic distributed data sets and each candidate subnetwork in a Spark cluster.

[0033] In addition, to achieve the above-mentioned purpose, the present invention also provides a model training device based on a Spark cluster, and the model training device based on a Spark cluster comprises:

[0034] An acquisition module, used to acquire training samples;

[0035] A generation module, configured to generate a plurality of candidate subnetworks through a deep neural subnetwork generator, and determine a training error corresponding to each candidate subnetwork based on the training sample, through a plurality of first elastic distributed data sets in a Spark cluster and each candidate subnetwork;

[0036] A determination module is used to determine an optimal sub-network among each candidate sub-network based on each of the training errors, and to determine a target model s based on the optimal sub-network and the main network.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a model training device based on a Spark cluster, and the model training device based on a Spark cluster includes: a memory, a processor, and a model training program based on a Spark cluster stored in the memory and executable on the processor, and the model training program based on the Spark cluster implements the steps of the aforementioned model training method based on the Spark cluster when executed by the processor.

[0038] In addition, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a model training program based on a Spark cluster is stored. When the model training program based on a Spark cluster is executed by a processor, the steps of the aforementioned model training method based on a Spark cluster are implemented.

[0039] The present invention obtains training samples; then generates multiple candidate subnetworks through a deep neural subnetwork generator, and based on the training samples, determines the training errors corresponding to each candidate subnetwork through multiple first elastic distributed data sets in a Spark cluster and each candidate subnetwork; then determines the optimal subnetwork in each candidate subnetwork based on each training error, and determines the target model based on the optimal subnetwork and the main network, thereby realizing model training through elastic distributed data sets in a Spark cluster, eliminating the need for model training through a GPU server, and improving the efficiency of model training. At the same time, it can also realize intelligent training and optimization of different network models according to different training samples, without the need for manual training and optimization of different network models, saving a lot of human resources, reducing the workload of model training and tuning, and improving the training efficiency of network models. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a structural diagram of a model training device based on a Spark cluster in a hardware operating environment involved in an embodiment of the present invention;

[0041] Figure 2 This is a flow chart of the first embodiment of the model training method based on Spark cluster of the present invention;

[0042] Figure 3 This is a schematic diagram of the functional modules of an embodiment of a model training device based on a Spark cluster of the present invention.

[0043] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0045] like Figure 1 As shown, Figure 1 It is a structural diagram of a model training device based on a Spark cluster in a hardware operating environment involved in an embodiment of the present invention.

[0046] The Spark cluster-based model training device in the embodiment of the present invention may be a PC. Figure 1 As shown, the model training device based on the Spark cluster may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0047] Optionally, the model training device based on the Spark cluster may also include a camera, RF (Radio Frequency) circuit, sensor, audio circuit, WiFi module, etc. Among them, sensors include light sensors, motion sensors and other sensors. Of course, the model training device based on the Spark cluster may also be configured with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be described in detail here.

[0048] Those skilled in the art will understand that Figure 1 The Spark cluster-based model training device structure shown in the figure does not constitute a limitation on the Spark cluster-based model training device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0049] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a model training program based on a Spark cluster.

[0050] exist Figure 1In the Spark cluster-based model training device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the Spark cluster-based model training program stored in the memory 1005.

[0051] In this embodiment, the Spark cluster-based model training device includes: a memory 1005, a processor 1001, and a Spark cluster-based model training program stored on the memory 1005 and executable on the processor 1001, wherein the processor 1001 calls the Spark cluster-based model training program stored in the memory 1005 and executes the steps of the Spark cluster-based model training method in the following embodiments.

[0052] The present invention also provides a model training method based on Spark cluster, referring to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the model training method based on Spark cluster of the present invention.

[0053] In this embodiment, the Spark cluster-based model training method includes:

[0054] Step S100, obtaining training samples;

[0055] In this embodiment, the training samples can be stored in the HDFS distributed file system. The training samples include training data and verification data. Before model training, the training data and verification data for model training are determined in advance, and the training data and verification data are stored in HDFS. The training data is used for model training to obtain training results, and the training error is determined through the verification data and training results.

[0056] Step S200, generating multiple candidate sub-networks through a deep neural sub-network generator, and based on the training samples, determining the training error corresponding to each candidate sub-network through multiple first elastic distributed data sets in the Spark cluster and each candidate sub-network;

[0057] In this embodiment, when model training is required, multiple candidate subnetworks are generated by a deep neural subnetwork generator, wherein the number of layers of each subnetwork in the multiple candidate subnetworks is not limited, that is, the number of network layers in each candidate subnetwork is random, and the number of candidate subnetworks generated by the deep neural subnetwork generator each time is also random. Of course, in other embodiments, the number of candidate subnetworks can also be limited, for example, a preset number of candidate subnetworks are generated by the deep neural subnetwork generator each time. It should be noted that the deep neural subnetwork generator can be pre-set by the algorithm developer, or multiple deep neural subnetwork generators can be pre-set, and one of the multiple deep neural subnetwork generators is randomly selected to generate multiple candidate subnetworks.

[0058] Then, based on the training samples, the training errors corresponding to the candidate subnetworks are determined through multiple first elastic distributed data sets and each candidate subnetwork in the Spark cluster. Specifically, each first elastic distributed data set is assigned a corresponding candidate subnetwork, so that the first elastic distributed data set trains the training samples based on the network parameters of the candidate subnetwork to obtain the training errors corresponding to the candidate subnetworks.

[0059] Step S300, based on each of the training errors, determining an optimal sub-network among the candidate sub-networks, and determining a target model based on the optimal sub-network and the main network.

[0060] In this embodiment, when the training error is obtained, the optimal subnetwork is determined in each candidate subnetwork based on the training error. For example, the candidate subnetwork with the smallest training error is used as the optimal subnetwork, or the training error and the number of layers of each candidate subnetwork are weighted, and the candidate subnetwork with the smallest weight score after the weight calculation is used as the optimal subnetwork. Then, the target model is determined based on the optimal subnetwork and the main network, wherein, if only one search is required, the neural network structure obtained by embedding the optimal subnetwork into the main network is the target model; if multiple searches are required, it is determined whether the number of searches reaches the specified number of times. If so, the neural network structure obtained by embedding the optimal subnetwork into the main network is the target model, and then the network model can be automatically trained through machine learning. For different businesses, there is no need to manually train and optimize different network models. Different machine learning models can be obtained by spending a small amount of time on algorithm engineers, saving a lot of human resources and improving the training efficiency of the model. At the same time, model training is implemented in the Spark cluster through machine learning technology, which improves the efficiency of model training.

[0061] Further, in one embodiment, step S200 includes:

[0062] Step S210: creating the first elastic distributed dataset in the Spark cluster based on the number of candidate subnetworks;

[0063] Step S220: Based on the training samples, a training error corresponding to each candidate subnetwork is determined through multiple first elastic distributed data sets and each candidate subnetwork in the Spark cluster.

[0064] In this embodiment, when the target network parameters are obtained, a first elastic distributed dataset is created in the Spark cluster based on the number of candidate subnetworks, so that the number of first elastic distributed datasets is greater than or equal to the number of candidate subnetworks. Of course, it is preferred that the number of first elastic distributed datasets is equal to the number of candidate subnetworks, so that the first elastic distributed dataset corresponds one-to-one to the candidate subnetworks.

[0065] In this embodiment, by combining the Spark cluster with automatic machine learning technology, the CPU can be used in a distributed manner to complete the training of complex big data models, and the computing power of the existing Spark cluster can be fully utilized for automatic machine learning. This can effectively solve the problem of long-term training of large data models in scenarios without GPUs, and then effectively utilize the existing IT infrastructure to complete model training, thereby improving the training efficiency of network models.

[0066] The Spark cluster-based model training method proposed in this embodiment obtains training samples; then generates multiple candidate subnetworks through a deep neural subnetwork generator, and based on the training samples, determines the training errors corresponding to each candidate subnetwork through multiple first elastic distributed data sets in the Spark cluster and each candidate subnetwork; then determines the optimal subnetwork in each candidate subnetwork based on each training error, and determines the target model based on the optimal subnetwork and the main network, thereby realizing model training through elastic distributed data sets in the Spark cluster, and eliminating the need for model training through a GPU server, thereby improving the efficiency of model training. At the same time, it can also realize intelligent training and optimization of different network models according to different training samples, without the need for manual training and optimization of different network models, saving a lot of human resources, reducing the workload of model training and tuning, and improving the training efficiency of network models.

[0067] Based on the first embodiment, a second embodiment of the model training method based on Spark cluster of the present invention is proposed. In this embodiment, step S200 includes:

[0068] Step S210, determining target network parameters corresponding to each candidate subnetwork based on the network parameters of each candidate subnetwork and the main network parameters of the main network;

[0069] Step S220: Based on the training samples, a training error corresponding to each candidate subnetwork is determined through multiple first elastic distributed data sets in the Spark cluster and each target network parameter.

[0070] In this embodiment, after multiple candidate subnetworks are generated by a deep neural subnetwork generator, the network parameters of each candidate subnetwork and the main network parameters of the main network are obtained, wherein if the candidate subnetwork is currently generated for the first time, the main network does not have main network parameters, and based on the network parameters of each candidate subnetwork and the main network parameters, the target network parameters corresponding to each candidate subnetwork are determined. Specifically, the network parameters of the candidate subnetwork are added to the main network parameters respectively to obtain the target network parameters of the corresponding candidate subnetwork.

[0071] In this embodiment, when the target network parameters are obtained, based on the training samples, the training errors corresponding to each candidate subnetwork are determined through multiple first elastic distributed data sets in the Spark cluster and each target network parameter. Specifically, each first elastic distributed data set is assigned a corresponding candidate subnetwork, so that the first elastic distributed data set trains the training samples based on the target network parameters s to obtain the training errors corresponding to each candidate subnetwork.

[0072] Further, in one embodiment, step S220 includes:

[0073] Step S221, respectively allocating each target network parameter to each first elastic distributed data set in the Spark cluster, so that the first elastic distributed data set is trained based on the training sample through the target network parameter;

[0074] Step S222: acquiring training results corresponding to each first elastic distributed data set through the second elastic distributed data set, and determining training errors corresponding to each candidate sub-network based on the training results.

[0075] In this embodiment, when the target network parameters corresponding to each candidate subnetwork are obtained, each target network parameter is respectively assigned to the first elastic distributed dataset in the Spark cluster, so that each target network parameter is assigned to a unique first elastic distributed dataset. The first elastic distributed dataset trains the training samples through the target network parameters to obtain the training results corresponding to each target network parameter.

[0076] Then, the training results corresponding to each first elastic distributed dataset are obtained through the second elastic distributed dataset, and then the training errors corresponding to each candidate sub-network are determined based on the training results. Specifically, the training errors corresponding to each candidate sub-network are determined according to the training results and the verification data in the training samples. The second elastic distributed dataset can be created in the Spark cluster in advance, for example, the second elastic distributed dataset is created at the same time as the first elastic distributed dataset is created, and the second elastic distributed dataset is used to synchronize the training results and network parameters corresponding to the first elastic distributed dataset.

[0077] It should be noted that when the first elastic distributed dataset trains the training samples through the target network parameters, the first elastic distributed dataset trains the training samples through the target network parameters for a preset number of times, wherein the preset number of times can be reasonably set, and according to the characteristics of RDD, each first elastic distributed dataset is converted once for each training, and after the preset number of trainings, the network parameters in the final RDD (target RDD) derived from the conversion are the training results corresponding to each first elastic distributed dataset.

[0078] The Spark cluster-based model training method proposed in this embodiment determines the target network parameters corresponding to each candidate subnetwork based on the network parameters of each candidate subnetwork and the main network parameters of the main network; then, based on the training samples, the training errors corresponding to each candidate subnetwork are determined through multiple first elastic distributed data sets in the Spark cluster and the target network parameters. The training errors can be accurately obtained through the target network parameters, thereby improving the accuracy of the training errors, thereby improving the training efficiency of the network model and the prediction accuracy of the network model.

[0079] Based on the first embodiment, a third embodiment of the model training method based on Spark cluster of the present invention is proposed. In this embodiment, the step of determining the optimal subnetwork in each candidate subnetwork based on each training error includes:

[0080] Step S311, based on the network parameters, obtaining the penalty coefficient corresponding to the number of network layers of each candidate subnetwork;

[0081] Step S312, based on each penalty coefficient and training error, respectively determine the weight score corresponding to each candidate sub-network;

[0082] Step S313: taking the candidate subnetwork with the smallest weight score among the candidate subnetworks as the optimal subnetwork.

[0083] In this embodiment, a preset penalty coefficient corresponding to the number of network layers, or a mapping relationship between the number of network layers and the preset penalty coefficient can be pre-set. After obtaining the training errors corresponding to each candidate sub-network, the corresponding number of network layers is determined according to the network parameters of each candidate sub-network. According to the number of network layers and the mapping relationship, the penalty coefficient corresponding to the number of network layers of each candidate sub-network is determined.

[0084] Then, based on each penalty coefficient and the training error, the weight score corresponding to each candidate sub-network is determined respectively. Specifically, the weight of the penalty coefficient and the weight of the training error are obtained respectively, and the weight score corresponding to each candidate sub-network is determined based on the penalty coefficient and the training error, the weight of the penalty coefficient and the weight of the training error. For example, the weight score = the weight of the penalty coefficient * the penalty coefficient + the weight of the training error * the training error.

[0085] The Spark cluster-based model training method proposed in this embodiment obtains the penalty coefficient corresponding to the number of network layers of each candidate subnetwork based on the network parameters; then determines the weight score corresponding to each candidate subnetwork based on each penalty coefficient and the training error; and then uses the candidate subnetwork with the smallest weight score among the candidate subnetworks as the optimal subnetwork. By comprehensively considering the penalty coefficient and the training error, the determined candidate subnetwork is a candidate subnetwork with a simple network and a small training error, which improves the efficiency of model training, reduces the complexity of the network model, and at the same time, improves the prediction accuracy of the subsequent target model.

[0086] Based on the first embodiment, a fourth embodiment of the model training method based on a Spark cluster of the present invention is proposed. In this embodiment, the model training method based on a Spark cluster further includes:

[0087] Step S400, when a plurality of candidate sub-networks are generated by the deep neural sub-network generator, updating the number of searches;

[0088] The step of determining the target model based on the optimal sub-network and the main network comprises:

[0089] Step S321, determining whether the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network;

[0090] Step S322, if the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network, determining whether the number of searches reaches a preset number of searches;

[0091] Step S323, when the search times reaches the preset search times, the target model is determined based on the optimal sub-network and the main network.

[0092] In this embodiment, in order to improve the prediction accuracy of the target model, it is necessary to superimpose multiple optimal sub-networks. Therefore, when multiple candidate sub-networks are generated by the deep neural sub-network generator, the search times are updated. Specifically, the search times are increased by 1 to obtain the updated search times.

[0093] Then, determine whether the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network, wherein, when no sub-network is embedded in the main network, the training error of the main network is greater than the training error corresponding to any candidate sub-network, and when a sub-network is embedded in the main network, the training error of the main network can be obtained according to the previous training error.

[0094] If the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network, it is determined whether the number of searches reaches the specified number. If so, the neural network structure obtained by embedding the optimal sub-network into the main network is the target model, and the number of searches is reset at the same time, that is, the network parameters of the target model are the network parameters of the optimal sub-network plus the main network parameters of the main network.

[0095] Furthermore, in one embodiment, after step a, the method further includes:

[0096] When the search times do not reach the preset search times, the network parameters corresponding to the optimal sub-network and the main network parameters of the main network are used as the network parameters of the main network, and the process returns to the step of generating multiple candidate sub-networks through the deep neural sub-network generator.

[0097] In this embodiment, the number of searches does not reach the preset number of searches, and the model training does not reach the specified number of times. Therefore, the network parameters corresponding to the optimal sub-network and the main network parameters of the main network are used as the main network parameters of the main network. That is to say, the optimal sub-network is embedded in the main network, and the main network after embedding the optimal sub-network is used as the main network. Then, the main network parameters of the main network are updated, and the step of generating multiple candidate sub-networks through the deep neural sub-network generator is returned to realize cyclic training.

[0098] Furthermore, in one embodiment, after step S421, the following steps are further included:

[0099] If the training error corresponding to the optimal sub-network is greater than or equal to the training error corresponding to the main network, the process returns to the step of generating multiple candidate sub-networks through the deep neural sub-network generator.

[0100] In this embodiment, if the training error corresponding to the optimal sub-network is greater than or equal to the training error corresponding to the main network, the update of the current main network is abandoned, and the step of generating multiple candidate sub-networks through the deep neural sub-network generator is returned to execute, wherein the search times may be updated or not when generating multiple candidate sub-networks this time; or, if the search times reaches the preset search times at this time, the search times are not updated when generating multiple candidate sub-networks this time, and if the search times do not reach the preset search times, the search times are updated.

[0101] The Spark cluster-based model training method proposed in this embodiment determines whether the training error corresponding to the optimal subnetwork is smaller than the training error corresponding to the main network; then if the training error corresponding to the optimal subnetwork is smaller than the training error corresponding to the main network, determines whether the number of searches reaches a preset number of searches; when the number of searches reaches the preset number of searches, determines the target model based on the optimal subnetwork and the main network, and can obtain the target model through the number of searches, the optimal subnetwork and the main network, thereby obtaining the target prediction model through multiple searches, further improving the accuracy of target model prediction and improving the model training efficiency.

[0102] The embodiment of the present invention also provides a model training device based on Spark cluster, referring to Figure 3 , the Spark cluster-based model training device includes:

[0103] An acquisition module 100 is used to acquire training samples;

[0104] A generation module 200 is used to generate multiple candidate sub-networks through a deep neural sub-network generator, and based on the training samples, determine the training error corresponding to each candidate sub-network through multiple first elastic distributed data sets in a Spark cluster and each candidate sub-network;

[0105] The determination module 300 is used to determine the optimal sub-network among the candidate sub-networks based on the training errors, and determine the target model based on the optimal sub-network and the main network.

[0106] Optionally, the generating module 200 is further used for:

[0107] Determine target network parameters corresponding to each candidate subnetwork based on the network parameters of each candidate subnetwork and the main network parameters of the main network;

[0108] Based on the training samples, a training error corresponding to each candidate subnetwork is determined through multiple first elastic distributed data sets in a Spark cluster and each target network parameter.

[0109] Optionally, the generating module 200 is further used for:

[0110] The training results corresponding to each first elastic distributed data set are obtained through the second elastic distributed data set, and the training errors corresponding to each candidate sub-network are determined based on the training results.

[0111] Optionally, the determination module 300 is further configured to:

[0112] Based on the network parameters, obtaining a penalty coefficient corresponding to the number of network layers of each candidate subnetwork;

[0113] Based on each penalty coefficient and training error, the weight score corresponding to each candidate sub-network is determined respectively;

[0114] The candidate subnetwork with the smallest weight score among the candidate subnetworks is taken as the optimal subnetwork.

[0115] Optionally, the Spark cluster-based model training device further includes:

[0116] When generating multiple candidate sub-networks through the deep neural sub-network generator, update the search count;

[0117] The determination module 300 is further configured to:

[0118] Determine whether the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network;

[0119] If the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network, determining whether the number of searches reaches a preset number of searches;

[0120] When the number of searches reaches the preset number of searches, the target model is determined based on the optimal sub-network and the main network.

[0121] Optionally, the determination module 300 is further configured to:

[0122] When the search times do not reach the preset search times, the network parameters corresponding to the optimal sub-network and the main network parameters of the main network are used as the main network parameters of the main network, and the step of generating multiple candidate sub-networks through the deep neural sub-network generator is returned to be executed.

[0123] Optionally, the determination module 300 is further configured to:

[0124] If the training error corresponding to the optimal sub-network is greater than or equal to the training error corresponding to the main network, the process returns to the step of generating multiple candidate sub-networks through the deep neural sub-network generator.

[0125] Optionally, the generating module 200 is further used for:

[0126] Based on the number of candidate subnetworks, create the first elastic distributed dataset in the Spark cluster;

[0127] Based on the training samples, a training error corresponding to each candidate subnetwork is determined through multiple first elastic distributed data sets and each candidate subnetwork in a Spark cluster.

[0128] The methods executed by the above-mentioned program modules can refer to the various embodiments of the model training method based on Spark cluster of the present invention, which will not be repeated here.

[0129] In addition, an embodiment of the present invention further proposes a storage medium, on which a model training program based on a Spark cluster is stored. When the model training program based on a Spark cluster is executed by a processor, the steps of the model training method based on a Spark cluster as described above are implemented.

[0130] Among them, the method implemented when the Spark cluster-based model training program running on the processor is executed can refer to the various embodiments of the Spark cluster-based model training method of the present invention, which will not be repeated here.

[0131] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0132] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0133] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0134] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A model training method based on Spark cluster, characterized in that: The Spark cluster-based model training method comprises the following steps: Obtain training samples, where the training samples are stored in the HDFS distributed file system; Generate multiple candidate subnetworks through a deep neural subnetwork generator, and based on the training samples, determine the training error corresponding to each candidate subnetwork through multiple first elastic distributed data sets in a Spark cluster and each candidate subnetwork; Based on each of the training errors, determining an optimal sub-network among each of the candidate sub-networks, and determining a target model based on the optimal sub-network and the main network; The step of determining the training error corresponding to each candidate subnetwork based on the training sample through the first elastic distributed data set in the Spark cluster and each candidate subnetwork includes: Determine target network parameters corresponding to each candidate subnetwork based on the network parameters of each candidate subnetwork and the main network parameters of the main network; Allocating each target network parameter to each first elastic distributed data set in the Spark cluster, so that the first elastic distributed data set is trained based on the training sample through the target network parameter; Obtaining training results corresponding to each of the first elastic distributed data sets through the second elastic distributed data set, and determining a training error corresponding to each of the candidate subnetworks based on the training results; The step of determining the optimal sub-network among the candidate sub-networks based on the training errors comprises: Based on the network parameters of each candidate sub-network, the penalty coefficient corresponding to the number of network layers of each candidate sub-network is obtained; Based on each penalty coefficient and training error, the weight score corresponding to each candidate sub-network is determined respectively; The candidate subnetwork with the smallest weight score among the candidate subnetworks is taken as the optimal subnetwork.

2. The Spark cluster-based model training method according to claim 1, characterized in that: The Spark cluster-based model training method also includes: When generating multiple candidate sub-networks through the deep neural sub-network generator, update the search count; The step of determining the target model based on the optimal sub-network and the main network comprises: Determine whether the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network; If the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network, determining whether the number of searches reaches a preset number of searches; When the number of searches reaches the preset number of searches, the target model is determined based on the optimal sub-network and the main network.

3. The Spark cluster-based model training method according to claim 2, characterized in that: After the step of determining whether the number of searches reaches a preset number of searches, the method further includes: When the search times do not reach the preset search times, the network parameters corresponding to the optimal sub-network and the main network parameters of the main network are used as the main network parameters of the main network, and the step of generating multiple candidate sub-networks through the deep neural sub-network generator is returned to be executed.

4. The Spark cluster-based model training method according to claim 2, characterized in that: After the step of determining whether the training error corresponding to the optimal sub-network is smaller than the training error corresponding to the main network, the method further includes: If the training error corresponding to the optimal sub-network is greater than or equal to the training error corresponding to the main network, the process returns to the step of generating multiple candidate sub-networks through the deep neural sub-network generator.

5. The Spark cluster-based model training method according to any one of claims 1 to 4, characterized in that: The step of determining the training error corresponding to each candidate subnetwork based on the training sample through multiple first elastic distributed data sets and each candidate subnetwork in the Spark cluster includes: Based on the number of candidate subnetworks, create the first elastic distributed dataset in the Spark cluster; Based on the training samples, a training error corresponding to each candidate subnetwork is determined through multiple first elastic distributed data sets and each candidate subnetwork in a Spark cluster.

6. A model training device based on Spark cluster, characterized in that: The Spark cluster-based model training device includes: An acquisition module is used to acquire training samples, where the training samples are stored in the HDFS distributed file system; A generation module is used to generate multiple candidate subnetworks through a deep neural subnetwork generator, and based on the training samples, determine the training errors corresponding to each candidate subnetwork through multiple first elastic distributed data sets in a Spark cluster and each candidate subnetwork; the generation module is also used to determine the target network parameters corresponding to each candidate subnetwork based on the network parameters of each candidate subnetwork and the main network parameters of the main network; respectively assign each target network parameter to each first elastic distributed data set in the Spark cluster, so that the first elastic distributed data set can be trained based on the training samples through the target network parameters; obtain the training results corresponding to each first elastic distributed data set through a second elastic distributed data set, and determine the training errors corresponding to each candidate subnetwork based on the training results; A determination module is used to determine the optimal subnetwork among the candidate subnetworks based on the training errors, and determine the target model based on the optimal subnetwork and the main network; the determination module is also used to obtain the penalty coefficient corresponding to the number of network layers of each candidate subnetwork based on the network parameters of each candidate subnetwork; based on each penalty coefficient and the training error, respectively determine the weight score corresponding to each candidate subnetwork; and take the candidate subnetwork with the smallest weight score among the candidate subnetworks as the optimal subnetwork.

7. A model training device based on Spark cluster, characterized in that: The Spark cluster-based model training device includes: a memory, a processor, and a Spark cluster-based model training program stored in the memory and executable on the processor. When the Spark cluster-based model training program is executed by the processor, the steps of the Spark cluster-based model training method as described in any one of claims 1 to 5 are implemented.

8. A storage medium, characterized in that: The storage medium stores a Spark cluster-based model training program, and when the Spark cluster-based model training program is executed by the processor, the steps of the Spark cluster-based model training method described in any one of claims 1 to 5 are implemented.

Images